Is stopwatch benchmarking acceptable?

Question

Does anyone ever use stopwatch benchmarking, or should a performance tool always be used? Are there any good free tools available for Java? What tools do you use?

Answer 1

How do you address operating system scheduling when benchmarking?

Benchmark for long enough on a system which is representative of the machine you will be using. If your OS slows down your application, then that should be part of the result.

There is no point in saying, my program would be faster, if only I didn't have an OS.

If you are using Linux, you can use tools such as numactl, chrt and taskset to control how CPUs are used and the scheduling.

Answer 2

I think a key question is the complexity and length of time of the operation.

I sometimes even use physical stopwatch measurements to see if something takes minutes, hours, days, or even weeks to compute (I am working with an application where run times on the orders of several days are not unheard of, even if seconds and minutes are the most common time spans).

However, the automation afforded by calls to any kind of clock system on the computer, like the java millis call referred to in the linked article, is clearly superior to manually seeing how long something runs.

Profilers are nice, when they work, but I have had problems applying them to our application, which usually involves dynamic code generation, dynamic loading of DLLs, and work performed in the two built-in just-in-time-compiled scripting languages of my application. They are quite often limited to assuming a single source language and other unrealistic expectations for complex software.

Answer 3

It's totally valid as long as you measure large enough intervals of time. I would execute 20-30 runs of what you intend to test so that the total elapsed time is over 1 second. I've noticed that time calculations based off System.currentTimeMillis() tend to be either 0ms or ~30ms; I don't think you can get anything more precise than that. You may want to try out System.nanoTime() if you really need to measure a small time interval:

• documentation: http://java.sun.com/javase/6/docs/api/java/lang/System.html#nanoTime()
• SO question about measuring small time spans, since System.nanoTime() has some issues, too: How can I measure time with microsecond precision in Java?

Answer 4

Profilers can get in the way of timings, so I would use a combination of stopwatch timing to identify overall performance problems, then use the profiler to work out where the time is being spent. Repeat the process as required.

Answer 5

Stopwatch benchmarking is fine, provided you measure enough iterations to be meaningful. Typically, I require a total elapsed time of some number of single digit seconds. Otherwise, your results are easily significantly skewed by scheduling, and other O/S interruptions to your process.

For this I use a little set of static methods I built a long time ago, which are based on System.currentTimeMillis().

For the profiling work I have used jProfiler for a number of years and have found it very good. I have recently looked over YourKit, which seems great from the WebSite, but I've not used it at all, personally.

To answer the question on scheduling interruptions, I find that doing repeated runs until consistency is achieved/observed works in practice to weed out anomalous results from process scheduling. I also find that thread scheduling has no practical impact for runs of between 5 and 30 seconds. Lastly, after you pass the few seconds threshold scheduling has, in my experience, negligible impact on the results - I find that a 5 second run consistently averages out the same as a 5 minute run for time/iteration.

You may also want to consider prerunning the tested code about 10,000 times to "warm up" the JIT, depending on the number of times you expect the tested code to run over time in real life.

Answer 6

I always use stopwatch benchmarking as it is so much easier. The results don't need to be very accurate for me though. If you need accurate results then you shouldn't use stopwatch benchmarking.