How to profile pthread mutex in linux?

Question

I would like to know how to profile a pthread mutex to see if there are any locking contention points in my code. (who likes contentious code, right? :) I know how to do a more

Answer 1

You might want to also give Intel VTune a try. It would report Wait Time and Wait Count for each sync primitive with call stacks too. Look through this document to see if this is what you're looking for http://software.intel.com/sites/products/documentation/hpc/amplifierxe/en-us/2011Update/lin/start/analyzing_locks_linux.pdf

Answer 2

The valgrind tool drd lets you specify a limit on how long a lock should be waited on before reporting an error.

This site mentions drd and also mentions their own tool called mutrace which looks like the kind of tool you're after. It tells you:

• how many times a mutex was locked
• how many times the mutex owning thread changed
• how many times a mutex was contended (already locked when a lock request was made)
• various stats on the duration a mutex was locked for

e.g.

mutrace: 10 most contended mutexes:

 Mutex #   Locked  Changed    Cont. tot.Time[ms] avg.Time[ms] max.Time[ms]       Type
      35   368268      407      275      120,822        0,000        0,894     normal
       5   234645      100       21       86,855        0,000        0,494     normal
      26   177324       47        4       98,610        0,001        0,150     normal
      19    55758       53        2       23,931        0,000        0,092     normal
      53      106       73        1        0,769        0,007        0,160     normal
      25    15156       70        1        6,633        0,000        0,019     normal
       4      973       10        1        4,376        0,004        0,174     normal
      75       68       62        0        0,038        0,001        0,004     normal
       9     1663       52        0        1,068        0,001        0,412     normal
       3   136553       41        0       61,408        0,000        0,281     normal
     ...      ...      ...      ...          ...          ...          ...        ...

mutrace: Total runtime 9678,142 ms.

Answer 3

I found 'mutrace' while exploring for the same use case. I did not use it yet though. It sounded good as it would not affect the runtime much as in valgrind.

Answer 4

You didn't mention your OS. If it's linux, I've used LTT quite a bit for this sort of profiling. It can be useful in two modes:

1. performance profiling -- e.g. profile your system for N seconds of standard use and then analyze the data

2. exception profiling -- e.g. run it around the clock in "flight recorder" mode (record the last N seconds of activity) and on some exceptional condition trigger a stop to the flight recorder. Come in the next morning and analyze the data.

I've been using and seeing 'mutexes' as the plural of mutex for years now without complaint. ;>

Answer 5

Its been 6+ years, but for the reference I would like to add another link for off-cpu time tracing which supposed to show your mutex contention. http://www.brendangregg.com/offcpuanalysis.html