如何从 JUnit 测试内部运行 JMH？

Question

How can I run JMH benchmarks inside my existing project using JUnit tests? The official documentation recommends making a separate project, using Maven shade plugin, and launching JMH inside the main method. Is this necessary and why is it recommended?

Answer 1

I've been running JMH inside my existing Maven project using JUnit with no apparent ill effects. I cannot answer why the authors recommend doing things differently. I have not observed a difference in results. JMH launches a separate JVM to run benchmarks to isolate them. Here is what I do:

• Add the JMH dependencies to your POM:

   <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-core</artifactId> <version>1.21</version> <scope>test</scope> </dependency> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-generator-annprocess</artifactId> <version>1.21</version> <scope>test</scope> </dependency>

  Note that I've placed them in scope test .

  In Eclipse, you may need to configure the annotation processor manually. NetBeans handles this automatically.

• Create your JUnit and JMH class. I've chosen to combine both into a single class, but that is up to you. Notice that OptionsBuilder.include is what actually determines which benchmarks will be run from your JUnit test!

   import java.util.ArrayList; import java.util.List; import java.util.Random; import java.util.concurrent.TimeUnit; import org.junit.Test; import org.openjdk.jmh.annotations.*; import org.openjdk.jmh.infra.Blackhole; import org.openjdk.jmh.runner.Runner; import org.openjdk.jmh.runner.options.*; public class TestBenchmark { @Test public void launchBenchmark() throws Exception { Options opt = new OptionsBuilder() // Specify which benchmarks to run. // You can be more specific if you'd like to run only one benchmark per test. .include(this.getClass().getName() + ".*") // Set the following options as needed .mode (Mode.AverageTime) .timeUnit(TimeUnit.MICROSECONDS) .warmupTime(TimeValue.seconds(1)) .warmupIterations(2) .measurementTime(TimeValue.seconds(1)) .measurementIterations(2) .threads(2) .forks(1) .shouldFailOnError(true) .shouldDoGC(true) //.jvmArgs("-XX:+UnlockDiagnosticVMOptions", "-XX:+PrintInlining") //.addProfiler(WinPerfAsmProfiler.class) .build(); new Runner(opt).run(); } // The JMH samples are the best documentation for how to use it // http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/ @State (Scope.Thread) public static class BenchmarkState { List<Integer> list; @Setup (Level.Trial) public void initialize() { Random rand = new Random(); list = new ArrayList<>(); for (int i = 0; i < 1000; i++) list.add (rand.nextInt()); } } @Benchmark public void benchmark1 (BenchmarkState state, Blackhole bh) { List<Integer> list = state.list; for (int i = 0; i < 1000; i++) bh.consume (list.get (i)); } }

• JMH's annotation processor seems to not work well with compile-on-save in NetBeans. You may need to do a full Clean and Build whenever you modify the benchmarks. (Any suggestions appreciated!)

• Run your launchBenchmark test and watch the results!

   ------------------------------------------------------- TESTS ------------------------------------------------------- Running com.Foo # JMH version: 1.21 # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11 # VM invoker: /usr/lib/jvm/java-8-jdk/jre/bin/java # VM options: <none> # Warmup: 2 iterations, 1 s each # Measurement: 2 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 2 threads, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: com.Foo.benchmark1 # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 4.258 us/op # Warmup Iteration 2: 4.359 us/op Iteration 1: 4.121 us/op Iteration 2: 4.029 us/op Result "benchmark1": 4.075 us/op # Run complete. Total time: 00:00:06 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark Mode Cnt Score Error Units Foo.benchmark1 avgt 2 4.075 us/op Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.013 sec

• Runner.run even returns RunResult objects on which you can do assertions, etc.

Answer 2

Other example

@State(Scope.Benchmark)
@Threads(1)
public class TestBenchmark {

    @Param({"10","100","1000"})
    public int iterations;


    @Setup(Level.Invocation)
    public void setupInvokation() throws Exception {
        // executed before each invocation of the benchmark
    }

    @Setup(Level.Iteration)
    public void setupIteration() throws Exception {
        // executed before each invocation of the iteration
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @Fork(warmups = 1, value = 1)
    @Warmup(batchSize = -1, iterations = 3, time = 10, timeUnit = TimeUnit.MILLISECONDS)
    @Measurement(batchSize = -1, iterations = 10, time = 10, timeUnit = TimeUnit.MILLISECONDS)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    public void test() throws Exception {
       Thread.sleep(ThreadLocalRandom.current().nextInt(0, iterations));
    }


    @Test
    public void benchmark() throws Exception {
        String[] argv = {};
        org.openjdk.jmh.Main.main(argv);
    }

}