Java G1 GC Processing Reference objects works slow

Question

I have hit counter running on Java. It works 24 hours and hits pass at about 100 times per second. During the day GC processing time slowly rises from 20-60 ms till 10000-60000 ms and then drops to 20-60 ms. Such pattern is repeated from time to time. From GC logs I found that mostly all time GC spend for processing Reference objects (Ref Proc). So what's the causes of such long GC time could be?

Server: Amazon EC2 m1.small
OS: Ubuntu 10.04.3 LTS
Java: Oracle 1.7.0_07

Example of GC logs:

2012-09-13T16:51:20.091+0400: 167239.936: [GC pause (young), 62.58395400 secs]
...
[Other: 62489.7 ms]
    [Choose CSet:   0.0 ms]
    [Ref Proc: 62433.9 ms]
    [Ref Enq:   0.0 ms]
    [Free CSet:   0.7 ms]
[Eden: 200M(200M)->0B(199M) Survivors: 4096K->5120K Heap: 578M(1024M)->380M(1024M)]

Time - "Ref Proc" graph:

09:37:59 - 242.4 ms
09:38:50 - 226.0 ms
09:39:00 - 83.6 ms
...
11:45:22 - 451.8 ms
11:46:04 - 489.3 ms
11:46:46 - 505.6 ms
...
14:05:40 - 1027.3 ms
14:06:01 - 796.6 ms
14:06:35 - 1064.0 ms
...
15:44:32 - 1920.4 ms
15:45:04 - 2116.7 ms
15:45:39 - 2196.8 ms
...
16:19:07 - 3983.3 ms
16:19:43 - 4494.9 ms
16:20:16 - 4065.2 ms
...
16:33:11 - 7690.1 ms
16:33:50 - 8501.4 ms
16:34:28 - 8059.3 ms
...
16:47:14 - 51378.6 ms
16:49:11 - 57529.2 ms
16:51:20 - 62433.9 ms
16:53:00 - 46.1 ms
16:53:30 - 45.5 ms
16:54:03 - 45.0 ms
...
16:54:38 - 57.0 ms
16:55:09 - 20.9 ms
16:55:43 - 21.3 ms
...
16:09:45 - 134.3 ms
16:10:21 - 142.1 ms
16:10:58 - 147.5 ms
...
17:18:51 - 177.3 ms
17:19:27 - 135.8 ms
17:20:03 - 179.6 ms

I found in java sources parameter PrintReferenceGC. GC log showed next

[SoftReference, 0 refs, 0.0000050 secs]
[WeakReference, 6 refs, 0.0000030 secs]
[FinalReference, 113 refs, 0.0011180 secs]
[PhantomReference, 0 refs, 0.0000020 secs]
[JNI Weak Reference, 3.9010450 secs]

This is some problem of JNI Weak Reference.

Answer 1

Its important that all my runs were with Yourkit profiler. I installed fastdebug build of OpenJDK7. At this version there is a parameter -XX:+TraceReferenceGC. After running with this parameter gc log shown about 5000 some finalizers. This problem was resolved by turning off socket probes of Yourkit.

Answer 2

As a first step you can use VisualVM or any tool to analyze your heap usage pattern. You might have a solution by optimising your code, by checking hotspots of object creation and that should be a permanant solution.

If that is not possible you have to find the optimal heap size and a GC algorithm suiting your applications behaviour. This is going to be a trial and error method. You will definitly have to go through tuning doc. The tuning you do now might again become inefficient later when your application behaviour changes or load pattern changes.

Try and find the suitable collector for you. You will be tweaking some parameters like

    -XX:+UseConcMarkSweepGC        
    -XX:SurvivorRatio=10 
    -XX:TargetSurvivorRatio=90 
    -XX:MaxTenuringThreshold=30

by trial and error.