We are running grails and we are noticing multiple full garbage collections are needed to clear out the permgen space.
2013-06-06T16:11:27.016+0000: 32582.145: [
Let me do a clarification about Concurrent Mark Sweep and it's incremental mode algorithm. CMS incremental mode was introduced to avoid CPU starvation on single core servers while background GC is running. Usage of incremental CMS is discouraged.
Incrementally mode does not free memory incrementally, it just doing long sleeps during mark phase of mark-sweep algorithm.
-XX:+CMSPermGenSweepingEnabled
is deprecated and synonymous to -XX:+CMSClassUnloadingEnabled
Incremental mode somewhat hiders dead object detection and my be a problem.
In addition, if any of classes (to be unloaded) has finilazer this also could explain 2 collections (JVM cannot unload individual classes, whole classloader is unloaded, so any of its classes can prevent collection).
I would advise you to properly size heap and perm gen, and configure CMS to be more aggressive if you want to keep that level of heap utilization. In my blog you can find few advises for tuning CMS.
If however, you run time is actively producing and abandoning class loaders tuning GC may not be enough.
I was facing similar issues with automated tests (each test were loading same classes in dedicated class loader for sake of isolation). Test were generally unstable, throwing OOME in perm gen. Solution was to force JVM to clean perm gen by polluting it with garbage data (here is snippet of code). It was causing Full GC though - probably not something you would like to see.
BTW
There is also -XX:CMSClassUnloadingMaxInterval=N
flag which let JVM collect Perm gen only every Nth collection. But this is not your case, cause perm gen is full.