Why is clock_gettime so erratic?

Question

Intro

• Section Old Question contains the initial question (Further Investigation and Conclusion have been added since).

Answer 1

Well as CLOCK_THREAD_CPUTIME_ID is implemented using rdtsc it will likely suffer from the same problems as it. The manual page for clock_gettime says:

The CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID clocks are realized on many platforms using timers from the CPUs (TSC on i386, AR.ITC on Itanium). These registers may differ between CPUs and as a consequence these clocks may return bogus results if a process is migrated to another CPU.

Which sounds like it might explain your problems? Maybe you should lock your process to one CPU to get stable results?

Answer 2

When you have a highly skewed distribution that cannot go negative, you're going to see large discrepancies between mean, median, and mode. The standard deviation is fairly meaningless for such a distribution.

It's usually a good idea to log-transform it. That will make it "more normal".