The way to solve cpu load too high of Java application

问题

Today, I found the cpu of load of my server is too high，and the server is just running a Java application.

Here are my operation steps.

1. I used top command to find the application's pid. The pid is 25713.

2. I used top -H -p 25713 command to find some pids which used the most of cpu. Such as 25719 tomcat 20 0 10.6g 1.5g 13m R 97.8 4.7 314:35.22 java.

3. I used jstack -F 25713 command to print the dump info.Such as "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f5f10021800 nid=0x6477 runnable

4. I searched the pid from the dump file. Then I found that the pids which used most of cpu are all like "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f5f10021800 nid=0x6477 runnable

5. After I used the jstack command, then the cpu became normal!

Here are my questions:

1. Why GC Threads made the cpu load too high.
2. Why after I used jstack command the cpu became nomal.

More than this time, every time.

Here are some normal logs.2015-10-10T10:17:52.019+0800: 71128.973: [GC (Allocation Failure) 2015-10-10T10:17:52.019+0800: 71128.973: [ParNew: 309991K->206K(348416K), 0.0051145 secs] 616178K->306393K(1009920K), 0.0052406 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]

When the CPU comes too high, the GC log stay in [GC (Allocation Failure) 2015-10-10T10:18:10.564+0800: 71147.518: [ParNew:, and there is no other logs.

When I execute jstack command, the log printed

2015-10-10T10:17:50.757+0800: 53501.137: [GC (Allocation Failure) 2015-10-10T10:17:50.757+0800: 53501.137: [ParNew: 210022K->245K(235968K), 369.6907808 secs] 400188K->1
90410K(1022400K), 369.6909604 secs] [Times: user=3475.15 sys=11.69, real=369.63 secs] 

回答1:

Just guessing, you might be affected by the futex_wait bug present in certain kernel versions.

More generally, jstack -F sends a signal to the process, which will interrupt any thread that may be sleeping. So maybe GC threads just spin-waiting for another thread that somehow missed a wakeup. I.e. if it's indeed stuck in a GC and sending a signal fixes the problem then this may point to a locking or memory ordering bug, if not in the kernel then in the JVM.

Instead of using jstack -F you could try sending SIGBREAK to the process and see if that has the same effect.

回答2:

why GC Threads made the cpu load too high.

Your JVM was probably running a full GC. And since your JVM is probably running with a huge heap (hinted by the 10.6 Gigabyte memory size), that is going to take a long time. It is also possible that your system is thrashing virtual memory.

why after i used jstack command the cpu became nomal.

Coincidence ... probably. The GC finished while you were running jstack.

---

If you want to investigate this, I suggest that you turn on garbage collection logging, and try to correlate the periods of high CPU load with the GC activity.

The other thing that the GC logs should tell you is whether your Tomcat's heap is too full. If your webapps have memory leaks, then that will cause the heap to fill up with objects that cannot be garbage collected. In time, that will lead to the JVM spending more and more time running the GC. If that is the problem, then you need to find and fix the memory leaks.

来源：https://stackoverflow.com/questions/33049375/the-way-to-solve-cpu-load-too-high-of-java-application