JMH: Using the same static object in all Benchmark tests

Question

I have a class that constructs some complicated data (imagine a large XML or JSON structure - that sort of thing). Constructing it takes time. So I want to construct it once

Answer 1

Unfortunately, JMH provides no way to share data between the benchmarks.

For one thing, this breaks benchmark isolation, when one benchmark can silently modify the input data for another benchmark, rendering the comparison incorrect. This is why you are required to @Setup the @State object for every benchmark.

But more importantly, whatever trick you build to share the data between the benchmarks (e.g. static field accessible from both) would break in the default "forked" mode, when JMH will execute each test in its own VM. Notably, the thing you are suggesting with static final Data TestSet.PARSE_ME would actually execute for each @Benchmark, since every new VM instance would have to initialize TestSet anyhow ;) Granted, you may disable forking, but that introduces more problems than it solves.

Therefore, it may be a better idea to invest time into making the setup costs more tolerable, so that it is not excruciatingly painful. E.g., deserialize the data from disk instead of computing it. Or, just come up with a faster way to compute.

Answer 2

I would just move all of your benchmark methods into one class, define your state object as an inner class, and inject the state to each of them:

public class ParserBenchmarks {

  @State(Scope.Thread)
  public static class StateHolder {
    Parser parser = null;

    @Setup(Level.Iteration)
    public void setup()
    {
      parser = new Parser(TestSet.PARSE_ME);
    }

    public Parser getParser() {
      return parser;
    }
  }

  @Benchmark
  public int getId_123(StateHolder stateHolder) {
    return stateHolder.getParser().getId(123);
  }

  @Benchmark
  public int getId_456(StateHolder stateHolder) {
    return stateHolder.getParser().getId(456);
  }

}

Note that all of your benchmark methods should return values otherwise the compiler could eliminate it as dead code.