Introduction to JMH

JMH is a tool set dedicated to code micro benchmarking. JMH is developed by the team implementing Java virtual machine. Since the JVM has become more and more intelligent, it may be optimized to varying degrees in the compilation stage, class loading stage and running stage of java files, so the code written by developers may not have the same performance as they expected.

JMH quick start

The most common is the performance comparison between ArrayList and LinkedList. We conduct 1000 groups of tests respectively, and each group of tests will execute 1000000 add calls to the List.

Test with main method

public class ArrayListVsLinkedList {
    private final static String DATA_STRING = "DUMMY DATA";
    private final static int MAX_CAPACITY = 1_000_000;
    private final static int MAX_ITERATIONS = 1000;

    private static void test(List<String> list) {
        for (int i = 0; i < MAX_CAPACITY; i++) {
            list.add(DATA_STRING);
        }
    }

    private static void arrayListPerfTest(int iterations) {
        final List<String> list = new ArrayList<>();
        final StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        test(list);
        stopWatch.stop();
        System.out.println(stopWatch.prettyPrint());
    }

    private static void linkListPerfTest(int iterations) {
        final List<String> list = new LinkedList<>();
        final StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        test(list);
        stopWatch.stop();
        System.out.println(stopWatch.prettyPrint());
    }

    public static void main(String[] args) {
        arrayListPerfTest(MAX_ITERATIONS);
        System.out.println(Strings.repeat('#',100));
        linkListPerfTest(MAX_ITERATIONS);
    }
}

At first glance, the performance of the add method of ArrayList is better than that of the add method of LinkedList. In fact, the random write performance of ArrayList is better than that of LinkedList (especially when ArrayList does not expand and copy the internal array). Due to the design of linked list, the delete operation performance of LinkedList will be better than that of ArrayList. We will have several problems in this test:

• Using StopWatch for time calculation actually records the number of nanoseconds at the beginning of method execution in StopWatch. This operation will essentially lead to a waste of CPU time.
• During the running of the code, the JVM may be optimized, such as loop expansion, runtime compilation, etc., which will lead to the participation of a group of non optimized performance data in statistics.

Micro benchmark using JMH

		<dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-core</artifactId>
            <version>1.19</version>
        </dependency>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>1.19</version>
            <scope>provided</scope>
        </dependency>

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Thread)
public class JMHExample1 {
    private final static String DATA_STATE = "DUMMY DATA";
    private List<String> arrayList;
    private List<String> linkedList;

    @Setup(Level.Iteration)
    public void setup() {
        arrayList = new ArrayList<>();
        linkedList = new LinkedList<>();
    }

    @Benchmark
    public List<String> arrayListAdd() {
        arrayList.add(DATA_STATE);
        return arrayList;
    }

    @Benchmark
    public List<String> linkedListAdd() {
        linkedList.add(DATA_STATE);
        return linkedList;
    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder().include(JMHExample1.class.getSimpleName()).forks(1).measurementIterations(10).warmupIterations(10).build();
        new Runner(opts).run();
    }
}

Run the main method to get the following output:

The conclusion is that the performance of the add method of ArrayList is better than that of LinkedList. Using JMH is obviously more rigorous.

@Benchmark mark benchmark method

If no method of a class is marked by @ Benchmark, the Benchmark will report an error.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Thread)
public class JMHExample2 {
    public void normalMethod() {

    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder().include(JMHExample2.class.getSimpleName()).forks(1).measurementIterations(10).warmupIterations(10).build();
        new Runner(opts).run();
    }
}

The results are as follows:

@Warmup and @ Measurement

These two annotations can act on both classes and methods. When there are both classes and methods, the annotation on the method will overwrite the one on the class. When the code in Option sets these two fields, the priority is the highest, and the annotation will become invalid.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Thread)
@Measurement(iterations = 5)
@Warmup(iterations = 2)
public class JMHExample3 {
    @Benchmark
    public void test() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @Measurement(iterations = 10)
    @Warmup(iterations = 3)
    @Benchmark
    public void test2() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder()
                .include(JMHExample3.class.getSimpleName())
                .forks(1)
//                .measurementIterations(10)
//                .warmupIterations(10)
                .build();
        new Runner(opts).run();
    }
}

The result you can see at this time is that the method overrides the class.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Thread)
@Measurement(iterations = 5)
@Warmup(iterations = 2)
public class JMHExample3 {
    @Benchmark
    public void test() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @Measurement(iterations = 10)
    @Warmup(iterations = 3)
    @Benchmark
    public void test2() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder()
                .include(JMHExample3.class.getSimpleName())
                .forks(1)
                .measurementIterations(10)
                .warmupIterations(10)
                .build();
        new Runner(opts).run();
    }
}

At this time, you can see that options has the highest priority. Explain the relevant output parameters

Four benchmarkmodes

    Throughput("thrpt", "Throughput, ops/time"),
    AverageTime("avgt", "Average time, time/op"),
    SampleTime("sample", "Sampling time"),
    SingleShotTime("ss", "Single shot invocation time"),
    All("all", "All benchmark modes");

There are four types:

type	explain
Throughput	throughput
AverageTime	Average response time
SampleTime	Time sampling
SingleShotTime	Cold test
all	The first four

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Thread)
@Measurement(iterations = 1)
@Warmup(iterations = 1)
public class JMHExample4 {
    @BenchmarkMode(Mode.AverageTime)
    @Benchmark
    public void testAverageTime() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @BenchmarkMode(Mode.Throughput)
    @Benchmark
    public void testThroughput() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @BenchmarkMode(Mode.SampleTime)
    @Benchmark
    public void testSampleTime() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @BenchmarkMode(Mode.SingleShotTime)
    @Benchmark
    public void testSingleShotTime() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @BenchmarkMode({Mode.AverageTime,Mode.Throughput})
    @Benchmark
    public void testAverageTimeAndThroughput() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    @BenchmarkMode(Mode.All)
    @Benchmark
    public void testAll() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder()
                .include(JMHExample4.class.getSimpleName())
                .forks(1)
                .build();
        new Runner(opts).run();
    }
}

Use of three states

public enum Scope {

    Benchmark
    Group,
    Thread,

}

type	describe
Benchmark thread sharing
Group	Thread group sharing, such as multithreading, two methods operate on a variable
Thread	Each thread holds a separate object

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@Measurement(iterations = 5)
@Warmup(iterations = 10)
@Threads(5)
@Fork(1)
public class JMHExample6 {
    @State(Scope.Thread)
    //Each thread will be created. You can see create instance five times
    public static class Test {
        public Test() {
            System.out.println("create instance");
        }

        public void method() {

        }
    }

    @Benchmark
    public void test(Test test) {
        test.method();
    }

    public static void main(String[] args) throws RunnerException {
        Options opts = new OptionsBuilder()
                .include(JMHExample6.class.getSimpleName())
                .forks(1)
                .build();
        new Runner(opts).run();
    }
}