A Brief Introduction to JMH

What is JMH

JMH is the abbreviation of Java Microbenchmark Harness. The Chinese meaning is roughly "JAVA microbenchmark suite". First, understand what a "benchmark" is. Baidu Encyclopedia defines it as follows:

Benchmarking It refers to the quantitative and comparable testing of a certain performance index of a kind of test object through the design of scientific testing methods, testing tools and testing systems.

Can be simply analogized to our computer commonly used master Lu, or mobile phone commonly used scoring software such as Antu Rabbit performance testing software. It is based on a certain benchmark or under specific conditions to test the performance of an object, such as graphics card, IO, CPU and so on.

Why use JMH

The characteristics of benchmarking are as follows:

(1) Repeatability: Repeatability testing can be carried out, which is conducive to comparing the results of each test, to get the long-term trend of performance results, and to provide reference for system optimization and capacity planning before on-line.

(2) Observability: Through all-round monitoring (including the beginning and end of testing, execution machine, server, database), timely understanding and analysis of what happened in the testing process.

(3) Demonstrability: Relevant personnel can intuitively and clearly understand the test results (web interface, dashboard, polygraph tree chart and other forms).

(4) Authenticity: The test results reflect the real situation that customers experience (true and accurate business scenarios + configuration consistent with production + reasonable and correct test methods).

(5) Executability: Relevant personnel can quickly test, verify, modify and tune (locatable and analyzable).

So it is very complicated and difficult to do a benchmark test that conforms to the characteristics. External factors can easily affect the final test results. Especially for JAVA benchmarking.


Some articles will tell us that JAVA is written in C++. Generally speaking, the program written in JAVA is unlikely to run more efficiently than the code written in C++. But JAVA does run more efficiently in some scenarios than C++. Don't think it's fantastic. In fact, JVM has become very intelligent with the development of these years, it will continue to optimize during operation.


This is good for our program, but it's a headache for performance testing. The number of times you run and the time you run may result in different results. It's hard to get a more stable result. In this case, there is a solution that is a large number of repeated calls, and a certain amount of pre-heating before the real test, so that the results are as accurate as possible.

In addition to these, we need a good demonstration of the results, so that we can judge the performance of the results through these demonstrations.

And these JMH have! A kind of

How to use JMH

Next, we use JMH as a benchmark test by taking several methods of string stitching as examples.

1. Import dependency

JMH comes with JDK9. If you are a pre-JDK9 version, you can also import openjdk.

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.19</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.19</version>
</dependency>

2. Directory structure

.
├── pom.xml
└── src
   ├── main
   │  └── java
   │     └── cn
   │        └── coder4j
   │           └── study
   │              └── demo
   │                 └── jmh
   │                    ├── benchmark
   │                    │  └── StringConnectBenchmark.java
   │                    └── runner
   │                       └── StringBuilderRunner.java
   └── test
      └── java
         └── cn
            └── coder4j
               └── study
                  └── demo

3. Specific code

• StringBuilderRunner.java

/**
 * coder4j.cn
 * Copyright (C) 2013-2018 All Rights Reserved.
 */
package cn.coder4j.study.demo.jmh.runner;

import cn.coder4j.study.demo.jmh.benchmark.StringConnectBenchmark;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

/**
 * @author buhao
 * @version StringBuilderRunner.java, v 0.1 2018-12-25 09:53 buhao
 */
public class StringBuilderRunner {

    public static void main( String[] args ) throws RunnerException {
        Options opt = new OptionsBuilder()
                // Import the class to be tested
                .include(StringConnectBenchmark.class.getSimpleName())
                // Preheat 5 wheels
                .warmupIterations(5)
                // Measure 10 rounds
                .measurementIterations(10)
                .mode(Mode.Throughput)
                .forks(3)
                .build();

        new Runner(opt).run();


    }

}

• StringConnectBenchmark.java

/**
 * coder4j.cn
 * Copyright (C) 2013-2018 All Rights Reserved.
 */
package cn.coder4j.study.demo.jmh.benchmark;

import org.openjdk.jmh.annotations.Benchmark;

/**
 * @author buhao
 * @version StringConnectBenchmark.java, v 0.1 2018-12-25 09:29 buhao
 */
public class StringConnectBenchmark {

    /**
     * String Builder benchmark for string splicing
     */
    @Benchmark
    public void testStringBuilder() {
        print(new StringBuilder().append(1).append(2).append(3).toString());
    }

    /**
     * Direct Addition Benchmark for String Mosaics
     */
    @Benchmark
    public void testStringAdd() {
        print(new String()+ 1 + 2 + 3);
    }

    /**
     * String Concat benchmark for string splicing
     */
    @Benchmark
    public void testStringConcat() {
        print(new String().concat("1").concat("2").concat("3"));
    }

    /**
     * String Buffer benchmark for string splicing
     */
    @Benchmark
    public void testStringBuffer() {
        print(new StringBuffer().append(1).append(2).append(3).toString());
    }

    /**
     * String Format benchmark for string splicing
     */
    @Benchmark
    public void testStringFormat(){
        print(String.format("%s%s%s", 1, 2, 3));
    }

    public void print(String str) {

    }
}

4. Operation results

# Run progress: 93.33% complete, ETA 00:00:15
# Fork: 3 of 3
objc[12440]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/bin/java (0x106a7d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x106af74e0). One of the two will be used. Which one is undefined.
# Warmup Iteration   1: 747281.755 ops/s
# Warmup Iteration   2: 924220.081 ops/s
# Warmup Iteration   3: 1129741.585 ops/s
# Warmup Iteration   4: 1135268.541 ops/s
# Warmup Iteration   5: 1062994.936 ops/s
Iteration   1: 1142834.160 ops/s
Iteration   2: 1143207.472 ops/s
Iteration   3: 1178363.827 ops/s
Iteration   4: 1156408.897 ops/s
Iteration   5: 1123123.829 ops/s
Iteration   6: 1086029.992 ops/s
Iteration   7: 1108795.147 ops/s
Iteration   8: 1125522.731 ops/s
Iteration   9: 1120021.744 ops/s
Iteration  10: 1119916.181 ops/s


Result "cn.coder4j.study.demo.jmh.benchmark.StringConnectBenchmark.testStringFormat":
  1132633.183 ±(99.9%) 16252.303 ops/s [Average]
  (min, avg, max) = (1082146.355, 1132633.183, 1182418.648), stdev = 24325.684
  CI (99.9%): [1116380.879, 1148885.486] (assumes normal distribution)


# Run complete. Total time: 00:03:57

Benchmark                                  Mode  Cnt          Score         Error  Units
StringConnectBenchmark.testStringAdd      thrpt   30   63728919.269 ±  906608.141  ops/s
StringConnectBenchmark.testStringBuffer   thrpt   30  112423521.098 ± 1157072.848  ops/s
StringConnectBenchmark.testStringBuilder  thrpt   30  110558976.274 ±  654163.111  ops/s
StringConnectBenchmark.testStringConcat   thrpt   30   44820009.200 ±  524305.660  ops/s
StringConnectBenchmark.testStringFormat   thrpt   30    1132633.183 ±   16252.303  ops/s

5. Code parsing

• StringBuilderRunner

The purpose of this runner class is to start benchmarking.


JMH usually starts in two ways, one is through the command line using the maven command. This is suitable for large benchmarks, such as those that run many, many times and for a long time. You can type a jar package directly, send it to the server, knock a command without paying attention to it, and come back in a few minutes, hours, days to see the results.

But in many cases, we just want to test a small function, there is no need to run a server. So JMH also provides a way to run through the Main method, as shown in the code above.

In the Main method, run the instance of org.openjdk.jmh.runner.Runner through the class org.openjdk.jmh.runner.options.Options. The focus here is on the construction of Options objects. Officially, an OptionsBuilder object is provided to build. This Builder object is streamed. Its common methods and corresponding annotations are as follows:

• StringConnectBenchmark

This is the class that actually executes the benchmark. This class is very similar to the class of unit test. Each test method contains the test code you want to execute. Just replace @Test with the @Benchmark annotation.


When the Main method of the Runner class runs, it finds these annotated methods, and then benchmarks them according to the specified rules. Of course, different methods sometimes require different rules, which can be specified separately in the form of annotations corresponding to the above methods.

6. Result analysis

The results were divided into three parts.


The first part is "Warmup Iteration...". This form of content. This shows the results of each preheating iteration.


The other part is "Iteration...". Formal content, which indicates the results of each benchmark iteration.

The last part is "Result...". This is the final result of all iterations. The first result tells us the information of maximum, minimum and average.


The final form structure information is the focus of our analysis, but its output is somewhat misplaced. At first, I have been struggling with what Error stands for (+906608.141). google found that Error actually does not output anything, and Score is 63728919.269 (+906608.141). I used the table to lay out the tables, explaining as follows:

Conclusion:

StringBuffer >= StringBuilder > String Direct Addition > StringConcat > > StringFormat

It can be seen that StringBuffer and StringBuilder have roughly the same performance, which are several orders of magnitude higher than the direct addition, and the direct addition is almost the same as the Concat method. But either of them is N orders of magnitude higher than StringFormat. So String's Format method must be carefully used, not used, disabled!!!
 

Related links

Reference link

1. openjdk official DEMO
2. openjdk official DEMO (translated version)
3. Talking about Benchmark Test
4. What is benchmarking
5. JMH Learning Notes _Very good
6. JAVA benchmarking using JMH
7. JMH does JAVA benchmarking

Code Links

1. DEMO code link