Why does park/unpark have 60% CPU usage?

Question

Recently we started stress testing our application (an XMPP based chat server) using YJP 11.0.9. During our test we noticed following strange behavior.

1. Sampling sh

Answer 1

High CPU time of Unsafe.unpark is a sign of excessive Context Switching. Here is an article to get the idea of how expensive is a context switch:

How long does it take to make a context switch?

The easiest option to find out CS count on Linux is run vmstat <seconds>.

Once high CS is confirmed (e.g. more 10K switches per core/per second) you take the offending thread (you can sort threads in YJP by their CPU time) and run strace -p <pid> -c to find out the cause of switching, e.g. thread is reading from socket in small pieces and get's switched off, in which case increasing socket buffer might help.

Answer 2

With certain low level blocking commands like read/write/park/lock the "CPU" time is over estimated as it assumes it is consuming CPU when actually the operation is blocking. The fact unpark/park are both high does suggest you have a problem, but I suspect you should take the lower of the two percentages as an estimate.