Memory leak in a Java web application

Question

I have a Java web application running on Tomcat 7 that appears to have a memory leak. The average memory usage of the application increases linearly over time when under load (d

Answer 1

I would suggest to use memoryAnalyzer for analyzing your heap, it gives far more information.
http://www.eclipse.org/mat/ there is a standalone application and eclipse embedded one. you just need to run jmap on your application and analyze the result with this.

Answer 2

I removed the following JMX configuration from tomcat\bin\setenv.bat:

set "JAVA_OPTS=%JAVA_OPTS% 
    -Dcom.sun.management.jmxremote=true 
    -Dcom.sun.management.jmxremote.port=9090
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false"

I can't get detailed memory heap dumps anymore, but the memory profile looks much better:

24 hours later, the memory profile looks the same:

Answer 3

I just encountered the same problem in a totally different application, so tomcat7 is probably not to blame. Memory Analyzer shows 10M unreachable String instances in the process (which has been running for about 2 months), and most/all of them have values that relate to Garbage Collection (e.g., "Allocation Failure", "end of minor GC")

Memory Analyzer

Full GC is now running every 2s but those Strings don't get collected. My guess is that we've hit a bug in the GC code. We use the following java version:

$ java -version
java version "1.7.0_06"
Java(TM) SE Runtime Environment (build 1.7.0_06-b24)
Java HotSpot(TM) 64-Bit Server VM (build 23.2-b09, mixed mode)

and the following VM parameters:

-Xms256m -Xmx768m -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC 
-XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:NewSize=32m -XX:MaxNewSize=64m
-XX:SurvivorRatio=8 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails 
-Xloggc:/path/to/file

Answer 4

By accident, I stumbled across the following lines in our Tomcat's conf/catalina.properties file that activate String caching. This might be related to your case if you have any of them turned on. It seems others are warning to use the feature.

tomcat.util.buf.StringCache.byte.enabled=true
#tomcat.util.buf.StringCache.char.enabled=true
#tomcat.util.buf.StringCache.trainThreshold=500000
#tomcat.util.buf.StringCache.cacheSize=5000

Answer 5

Try to use MAT and make sure that when you parse the heapdump, do it not dropping out the unreachable objects.

To do so, follow the tutorial here.

Then you can run a simple Mem Leak Analysis (This is a good tutorial)

That should quickly lead you to the root cause.

Answer 6

I can recommend jvisualvm which comes along with every Java installation. Start the programm, connect to your Webapplication. Go to Monitor -> Heap Dump. It now may take some time (depending on the size). The navigation through the Heap Dump is quite easy, but the meaning you have to figure out yourself (not too complicated though), e.g.

Go to Classes (within the heapdump), select java.lang.String, right click Show in Instances View. After that you'll see on the left side table String instances currently active in your system. Klick on one String instance and you'll see some String preferenes on the right-upper part of the right table, like the value of the String.

On the bottom-right part of the right table you'll see where this String instance is referenced from. Here you have to check where the most of your *String*s are being referenced from. But with your case (176/210, good propability to find some String examples which causes your problems soon) it should be clear after some inspection where the problem lies.