Java Algorithm time measure

Question

My problem is that when i try to get an accurate measurement of my algorithms execution time, the result differ to much once the first set of tests have been done.

I have 7 text files with Integers, each text file have a power of two elements.

2 ⁴ = 16
2 ⁸ = 256
2 ¹² = 4096
2 ¹⁶ = 65536
2 ²⁰ = 1048576
2 ²² = 4194304
2 ²⁴ = 16777216

I am running these tests X number of times to get the execution time. A test case is considered executing all the above tests once. And i am doing this several times without changing the text files data.

To measure my algorithm execution time, im using System.nanoTime();

        long start = System.nanoTime();
        (Algorithm)
        long elapsedNanoTime = System.nanoTime() - start;

However, the results, shows a major drop for the starting tests 16 and 256 after the first test case have been executed.

Test case, iteration 1:

16 elements takes 325336n
256 elements takes 414861n
4096 elements takes 2061728n
65536 elements takes 21111426n
1048576 elements takes 326898979n
4194304 elements takes 1487154649n
16777216 elements takes 6700800203n

Test case, iteration 2:

16 elements takes 2925n
256 elements takes 36864n
4096 elements takes 885603n
65536 elements takes 15839933n
1048576 elements takes 332000198n
4194304 elements takes 1484967410n
16777216 elements takes 6695675287n

Test case, iteration 3:

16 elements takes 2926n
256 elements takes 35985n
4096 elements takes 679635n
65536 elements takes 15462227n
1048576 elements takes 328179551n
4194304 elements takes 1483733064n
16777216 elements takes 6704160641n

And if i run each test case separately, "Compiling" the program to do only 7 tests, the results all comes out like the above iteration 1.

So do anyone have some insight as to why the execution time differ from the first test case and the others? Does it have to do with the initialization of the program or that the memory already have the allocated data somewhere? Because as of now, i'm not sure which execution time data is correct.

Thanks in advance.

Answer 1

Benchmarking is hard, and you've stumbled upon one of the more common troublesome cases; the JIT compiler . The JVM is actually responsible for compiling some of the code you run as its running , in order to optimize the bytecode further than the compiler itself can hope to do.

Describing how to ensure your benchmarks are accurate is beyond what can fit in an answer (but there are many resources ) but for this particular issue what you want to do is run your benchmark multiple times (in the same JVM) and discard the first few runs as noise. Once the benchmark has been run a couple of times the JIT will have had a chance to optimize your code for you, and likely won't make more optimizations in successive runs.

Answer 2

Consider using jmh as a framework for your micro benchmark. As pointed out out in @dimo414's answer, benchmarking is hard on the JVM.