permgen garbage collection takes multiple Full GC

Question

We are running grails and we are noticing multiple full garbage collections are needed to clear out the permgen space.

2013-06-06T16:11:27.016+0000: 32582.145: [         


        

Answer 1

Let me do a clarification about Concurrent Mark Sweep and it's incremental mode algorithm. CMS incremental mode was introduced to avoid CPU starvation on single core servers while background GC is running. Usage of incremental CMS is discouraged.

Incrementally mode does not free memory incrementally, it just doing long sleeps during mark phase of mark-sweep algorithm.

-XX:+CMSPermGenSweepingEnabled is deprecated and synonymous to -XX:+CMSClassUnloadingEnabled

Incremental mode somewhat hiders dead object detection and my be a problem.

In addition, if any of classes (to be unloaded) has finilazer this also could explain 2 collections (JVM cannot unload individual classes, whole classloader is unloaded, so any of its classes can prevent collection).

I would advise you to properly size heap and perm gen, and configure CMS to be more aggressive if you want to keep that level of heap utilization. In my blog you can find few advises for tuning CMS.

If however, you run time is actively producing and abandoning class loaders tuning GC may not be enough.

I was facing similar issues with automated tests (each test were loading same classes in dedicated class loader for sake of isolation). Test were generally unstable, throwing OOME in perm gen. Solution was to force JVM to clean perm gen by polluting it with garbage data (here is snippet of code). It was causing Full GC though - probably not something you would like to see.

BTW There is also -XX:CMSClassUnloadingMaxInterval=N flag which let JVM collect Perm gen only every Nth collection. But this is not your case, cause perm gen is full.

Answer 2

One way to identify what is being collected in that last FullGC collection is to print class histograms before/after Full GC: -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC.

This way you can compare histograms from all collections and identify which classes are collected in the last one.

To your second question considering PermGen, the usual advice is to identify the sufficient size of the PermGen for your application/workload and stick with that. You need to investigate why there is so many objects being placed into the PermGen.