Experiencing occasional long garbage collection delays, why?

Question

I\'m having a hard time dealing with a Java garbage collection problem, and interpreting the logs.

My application requires that no GC takes longer than 2 seconds, and id

Answer 1

I think you are having this UseConcMarkSweepGC and NewRatio bug. Since your new-space is no way near being one tenth of the -Jmx=9G. The bug includes a workaround (NewSize in absolute size).

Another flag that might be very important for you is CMSInitiatingOccupancyFraction. It is set at 92% in java6 and was 68% in java5. If the old-space grows larger the CMS threadpool will start to do their work. If you have CPU to spend, it is no danger to have a live-set that is above the initiating fraction.

It would be nice if you had included the GC stats after you fixed the memory paging problem.

Answer 2

Have you run your app through a profiler to see that what you think is happening with respect to memeory is what is actually happening?

One thing I do when I am looking into this is use the Netbeans profiler (though any profiler should give this to you) is look at the live bytes (the memory allocation) and see if the ones that have a large amount of allocated bytes and allocated objectd are what I am expecting and are coming from where I expect.

You can also probably use the profiler to look at the effect your tuning is having, run the app without any tuing args, and then add your tuning args and run it again and see what is going on with the memory.

Answer 3

If you have strict timing requirements, perhaps you should check into the Java Real-Time System.

RTSJ /Java RTS provides:

An API set, semantic Java VM enhancements, and JVM-to-OS layer modifications which allow Java developers to correctly reason about and control the temporal behavior of Java applications.

Answer 4

for me problem was full survivor spaces. As CMS isn't compactional, garbage was promoted directly to old gen. this could be very expensive with such large heap. It's necessary to increase Survivor spaces and MaxTenuringThreshold to avoid promotion as much as possible.

regards, Alex

Answer 5

From the times, it seems the GC actually doesn't run all the time (see the user times), so most of the time, it's waiting.

Just a wild guess: Doesn't it swap? How much memory does the machine have? How much of it does the java process get (resident set size)?

Edit: why is it waiting:

Look at this (from your transcript)

[Times: user=0.39 sys=0.01, real=12.96 secs]

This means that (I guess from the beginning to the end of GC), 12 (almost 13) seconds passed. Of these 13 seconds, .39 was spent running in user mode, .01 was spent running in kernel mode. If the time collection method isn't totally flawed (ie. the numbers really represent the running time of the GC process/thread), this means at least 12 seconds waiting.

Answer 6

It can be very hard to tell without actually seeing, and in some cases profiling the code.

Have you implemented finalize() for any of your objects? That will cause a big GC penalty. It would also be interesting to see a test run with a heap of maybe 6 Gigs, if you get a disproportionate improvement in performance it would indicate that the GC is thrashing for memory.