Benchmarking Java HashMap Get (JMH vs Looping)

Question

My ultimate goal is to create a comprehensive set of benchmarks for several Java primitive collection libraries using the standard Java collections as a baseline. In the past I

Answer 1

First, looping test measures average time, while your JMH code is configured for sample time. From the Mode.SampleTime javadoc:

Sample time: samples the time for each operation.

Individual executions of Map.get() are pretty fast to the point when the underlying time measurement system will report 0 for some of the executions due to time measurement granularity (read Nanotrusting the Nanotime blog post by the JMH author for more information).

In the sample mode the benchmarks collects individual sample times into an array and then calculates averages and percentiles using that array. When more then half of the array values are zero (in your particular setup more than 90% of array values are zero, as indicated by the p(90.0000) = 0.000 ns/op) the average is bound to be pretty low but when you see p(50) = 0 (and especially p(90) = 0) in your output the only conclusion you can reliable make is that these results are garbage and you need to find another way to measure that code.

• You should use Mode.AverageTime (or Mode.Throughput) benchmark mode. Leave Mode.SampleTime for situations when individual invocation takes substantial time.

• You could add a "baseline" benchmark which executes the if () and key++ in order to isolate the time required for key bookkeeping and the actual Map.get() time, but you will need to explain the results (the blog post linked above describes pitfalls with subtracting "baselines" from "real" measurements).

• You could try to use Blackhole.consumeCPU() to increase the execution time of individual invocation (see the previous point about "baselines" and associated pitfalls).