OutOfMemoryError when hot-deploying SpringMVC app to Tomcat7 - possible relation to log4j2

Question

I am having trouble Hot Deploying a Spring-MVC 4.0 (not SpringBoot) Web Application. I am trying to go xml-less and just use JavaConfig. OutOfMemoryErrors result when

Answer 1

Long time ago I have the same problem with tomcat7 hot deployment. When i start to using JAVA_OPTS with

-XX:+CMSClassUnloadingEnabled

solve my problem. I would like to explain, but i think that are better explanations.

What does JVM flag CMSClassUnloadingEnabled actually do?

http://frankkieviet.blogspot.ca/2006/10/classloader-leaks-dreaded-permgen-space.html

Answer 2

I believe Luke is right that this issue may be caused by a bug in Log4j2.

The bug is that if the <display-name> element is not present in the web.xml, the Log4jServletContextListener class responsible for log4j resource management can only start, but not stop log4j.

This means that no cleanup happens when the web application is stopped or restarted. JMX MBeans are not unregistered, but also any threads are not stopped, and as a result Log4j classes are not unloaded. The leaked memory grows every time the web app is restarted, because each web app has its own classloader (so the VM sees them as different classes).

This bug is fixed in Git master now and the fix will be part of the log4j 2.5 release. Meanwhile, please use a <display-name> element in your web.xml.

Answer 3

I believe that this is a bug in Log4j2.

I've been looking into similar PermGen memory leaks in an application I work on. By using a profiler I could see that when Log4j2 was shutting down it was not deregistering all of the JMX management MBeans it had previously registered. Because these MBeans were left in the JVM's internal MBean registry, the undeployed web application's classes couldn't be entirely removed from memory because there were references to instances of these classes.

Like you in your previous question, I too was seeing Log4j2 generate the message

No Log4j context configuration provided. This is very unusual.

during startup. In my case, however, the fix was somewhat odd - the web app didn't have a <display-name> set in our web.xml, and giving it a display name made that message and the memory leak go away!

I suspect that there is a bug in Log4j2 in that it doesn't deregister MBeans it registered if it is started without the context it mentions. Passing a display name is one way to create enough of a context, adding a context param for the location of the configuration appears to be another. I will look at raising an issue for this if there isn't already one.There is currently an issue on the Log4j2 JIRA that mentions problems if the display-name is missing, I've now added a comment to this issue to mention that there is also a memory leak in this situation.

As far as I see it, you have three options.

1. Live with a web.xml file. It's possible that it would only need to have a <display-name>.

2. Disable JMX in Log4j2 by setting the system property log4j2.disable.jmx to true. (This is best done with a command-line argument -Dlog4j2.disable.jmx=true.) This option is mentioned in the Log4j2 JMX documentation.

3. Look into Mattias Jiderhamn's ClassLoaderLeakPreventor library. One of its functions during web application shutdown is to deregister any MBeans that a web application registered but didn't deregister. You wouldn't be able to use the library as a JAR file directly because it needs to be added to web.xml as a listener, presumably for the benefit of those still using older Servlets versions. However, there are some options to get around this.

   Firstly, there is only one .java file in this library, and you could perhaps include its source (or just the parts of its source that unregister MBeans) in your application if you also add a @WebListener annotation to the class. Secondly, you could extend the ClassLoaderLeakPreventor class and add the @WebListener annotation to your subclass.