Performance profiling on Linux

Question

What are the best tools for profiling C/C++ applications on *nix?

(I\'m hoping to profile a server that is a mix of (blocking) file IO, epoll for network and fork()/

Answer 1

The FOSS answer, as already mentioned, is to build with -pg and then use gprof to analyse the output. If it's a product/project that justifies throwing some money at, I would also be tempted to use IBM/Rationals Quantify profiler as that makes it easier to view the profiling data, drill down to the line level or look at it in a '10000ft' level.

Of course there might be viewer for gprof available that can do the same thing, but I am not aware of any.

Answer 2

If you can take your application to freeBSD, OS X , or Solaris you can use dtrace, although dtrace is an analyst oriented tool -- i.e., you need to drive it -- read: script it. Nothing else can give you the level of granularity you need; Dtrace can not just profile the latencies of function calls in user-land; it can also follow a context switch into the kernel.

Answer 3

Allinea MAP is a profiler for C++ and other native languages on Linux. It is commercially supported by my employer. It has a graphical interface and source-line level profiling and profiles code with almost no slowdown which makes it very accurate where timing of other subsystems is relevant - such as for IO.

Callgrind has been useful and accurate - but the slowdown was ~5x so I could only do smaller runs. It can actually count the number of times a function is called which is useful for understanding asymptotic behavior.

Answer 4

As mentioned in the accepted answer, Zoom can do some amazing things. I've used it to understand thread behavior all the way down to optimizing the assembly generated by the compiler.

Answer 5

I recommend taking stackshots, for which pstack is useful. Here's some more information:

1. Comments on gprof.

2. How stackshots work.

3. A blow-by-blow example.

4. A very short explanation.

If you want to spend money, Zoom looks like a pretty good tool.

Answer 6

The canonical example of a full system profiling tool (for Solaris, OS X, FreeBSD) is DTrace. But it is not yet fully available on Linux (you can try here but the site is down for me at the moment, and I haven't tried it myself). There are many tools, in various states of usefulness, for doing full system profiling and kernel profiling on Linux.

You might consider investigating:

• oprofile
• SystemTap
• bootchart
• strace (e.g. this SO answer