Tomcat stopped without any log or any stack

Question

We have trouble with Tomcat 5.5 which stops at night on our production servers (Linux CentOS 4.8) and we have no idea why it stops... There is no Tomcat\'s log in catalina.out o

Answer 1

1) Make sure you know where stderr is redirected and check if anything got printed there.

2) Check the memory limits on Tomcat and how much free memory does the system have. Review the Linux system logs under /var/log to see if anything suspicious happened during the time. For example, kernel can randomly kill a process (almost) without a trace if the system is running low on memory.

We've ran 5.5 in production for years and never had any unexplained shutdowns, FWIW.

Answer 2

This worked for me.

As suggested here in other answers checked system logs in /var/log/messages but permission denied for me. So, I used dmesg command instead and got this in the logs

"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".

In the output I also noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.

After spending hours finally got the reason.

ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have been completed successfully and due to some reason the processes were not killed even after the shutdown, which was causing the high memory usage.

So, killed all running tomcat processes using kill. SWAP memory got freed.

Started tomcat again, worked fine. :)

Answer 3

Most probably there is a stack-overflow exception. This is typical behavior of Tomcat when it happens. For example, you're trying to serialize to JSON or XML beans with cyclic dependencies (but without handling of the cycles).

Everytime I had this issue (several times) it always has been this one. All other stops are usually logged properly (like OutOfMemory etc).

This type of stops leaves no trace anywhere.

Answer 4

Tomcat 7 has an option inside catalina to prevent the System.exit class call or something similar: http://ci.apache.org/projects/tomcat/tomcat7/docs/security-manager-howto.html .

Maybe there's a similar option for the 5.5 version. Try the documentation.

There are options to redirect the output to the same console that you use to start Tomcat. This information is redirected to logs when you execute on Unix based systems, on Windows, it remains with the console if not redirected.