JMH: Using the same static object in all Benchmark tests

Question

I have a class that constructs some complicated data (imagine a large XML or JSON structure - that sort of thing). Constructing it takes time. So I want to construct it once, and then use that same data in all tests. Currently I basically have a public static object instance defined in a class that defines main, and then refer to it explicitly in the tests (code is a very simplified example):

public class Data 
{
    // This class constructs some complicated data 
}

public class TestSet 
{
    public static final Data PARSE_ME = new Data(...);

    public static void main(String[] args) throws RunnerException 
    {
        Options opt = new OptionsBuilder()
                .include(".*ParserTest") // several tests
                .forks(1)
                .build();

        new Runner(opt).run();
    }
}

@State(Scope.Thread)
public class SomeParserTest
{
    @Setup(Level.Iteration)
    public void setup()
    {
        Parser parser = new Parser(TestSet.PARSE_ME);
    }

    @Benchmark
    public void getId() 
    {
        parser.getId(123);
    }
}

And this is awful of course... An equally evil option would be creating a separate class just so that it can hold a single static object. It would be nice to use something like

Options opt = new OptionsBuilder()
    ...
    .param(/*my Data object comes here*/)

but param only accepts Strings, so not sure how would I pass an object (and more importantly: the same instance of the object!) to it.

So is there anything more elegant, than a global object, I described above?

Answer 1

Unfortunately, JMH provides no way to share data between the benchmarks.

For one thing, this breaks benchmark isolation, when one benchmark can silently modify the input data for another benchmark, rendering the comparison incorrect. This is why you are required to @Setup the @State object for every benchmark.

But more importantly, whatever trick you build to share the data between the benchmarks (e.g. static field accessible from both) would break in the default "forked" mode, when JMH will execute each test in its own VM. Notably, the thing you are suggesting with static final Data TestSet.PARSE_ME would actually execute for each @Benchmark, since every new VM instance would have to initialize TestSet anyhow ;) Granted, you may disable forking, but that introduces more problems than it solves.

Therefore, it may be a better idea to invest time into making the setup costs more tolerable, so that it is not excruciatingly painful. E.g., deserialize the data from disk instead of computing it. Or, just come up with a faster way to compute.

Answer 2

I would just move all of your benchmark methods into one class, define your state object as an inner class, and inject the state to each of them:

public class ParserBenchmarks {

  @State(Scope.Thread)
  public static class StateHolder {
    Parser parser = null;

    @Setup(Level.Iteration)
    public void setup()
    {
      parser = new Parser(TestSet.PARSE_ME);
    }

    public Parser getParser() {
      return parser;
    }
  }

  @Benchmark
  public int getId_123(StateHolder stateHolder) {
    return stateHolder.getParser().getId(123);
  }

  @Benchmark
  public int getId_456(StateHolder stateHolder) {
    return stateHolder.getParser().getId(456);
  }

}

Note that all of your benchmark methods should return values otherwise the compiler could eliminate it as dead code.