How to troubleshoot Out Of Memory error on Production system

Question

We are using JBoss_4_0_4_GA with JDK 1.5.0 (no updates) on a Windows

The JBoss server is run within a Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org.

Since the JVM is so old I cannot even use the -XX:+HeapDumpOnOutOfMemoryError option on the JVM.

What are my option to find out the issue?

As usual the Out of Memory exception is happening on different parts of the application.

I do not have the liberty to upgrade the JVM right away.

    The current VM settings
    Java Additional Parameters
    wrapper.java.additional.1=-Xms512m
    wrapper.java.additional.2=-Xmx1024m
    wrapper.java.additional.3=-Dsun.rmi.dgc.client.gcInterval=3600000
    wrapper.java.additional.4=-Dsun.rmi.dgc.server.gcInterval=3600000
    wrapper.java.additional.5=-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
    wrapper.java.additional.6=-Djava.endorsed.dirs=D:/jboss-4.0.4.GA/lib/endorsed

Snippets of the exception

INFO | jvm 1 | 2012/05/31 11:25:03 | 11:25:03,502 ERROR [SOAPFaultExceptionHelper] SOAP request exception INFO | jvm 1 | 2012/05/31 11:25:03 | java.rmi.RemoteException: java.lang.OutOfMemoryError: Java heap space; nested exception is: INFO | jvm 1 | 2012/05/31 11:25:03 | java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

INFO | jvm 1 | 2012/05/31 11:25:03 | Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2012/05/31 11:25:03 | at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205) INFO | jvm 1 | 2012/05/31 11:25:03 | at java.util.concurrent.FutureTask.get(FutureTask.java:80)

Answer 1

You can try to do memory dumps with jmap (in tags you mentioned java 5 so this should be possible). Do several dumps when the server still works (like every hour or so).

Then analyze them in Eclipse MAT. Search for objects, or collections of objects that grow bigger on every dump. This will most probably be your memory leak.

Answer 2

I see two options, each of them having their pros and cons:

1. Install a profiler agent and connect a profiler on your application. This has of course some serious performance impact which may be not acceptable in your production environment. But this would allow you to monitor the application and perform a memory dump when Heap space is almost full.

2. Reproduce your production environment elsewhere and also use a profiler. If your issues only appear with big loads, you may have to create load tests so that you reach the OOME.

Using a profiler is your best chance to find memory leaks.

Otherwise, you can always try to perform static code reviewing, using PMD and Findbugs (amongst other static analysis tools) which may spot some mistakes.