Thread Utilization profiling on linux

Question

Linux perf-tools are great for finding hotspots in CPU cycles and optimizing those hotspots. But once some parts are parallelized it becomes difficult to spot the sequential par

Answer 1

You can get the result you want using a great tool we use to analyze Off-CPU Analysis - Off-CPU Flame Graphs which is apart of Flame Graphs

I used the Off-CPU analysis

Off-CPU analysis is a performance methodology where off-CPU time is measured and studied, along with context such as stack traces. It differs from CPU profiling, which only examines threads if they are executing on-CPU.

This tool is based on the tools you mentioned as the preferred ones - perf, bcctools, however, it provides a really easy to use output called flame graph which interactive SVG file looks like this SVG Off-CPU Time Flame Graph.

The width is proportional to the total time in the code paths, so look for the widest towers first to understand the biggest sources of latency. The left-to-right ordering has no meaning, and the y-axis is the stack depth.

2 more helpful analysis which are part of the Off-CPU Flame Graphs can also help you - Personally, I did not tried them.

Wakeup

This lets us solve more problems than off-CPU tracing alone, as the wakeup information can explain the real reason for blocking.

And Chain Graph

Chain graphs are an experimental visualization that associates off-CPU stacks with their wakeup stacks

There is also an experimental visualization which combines both CPU and Off-CPU flame graphs Hot/Cold Flame Graphs

This shows all thread time in one graph, and allows direct comparisons between on- and off-CPU code path durations.

It requires a little time to read about this profiling tool and understands its concepts, however, using it is super easy and its output is easier to analyze than other tools you mentioned above.

Good Luck!