how to properly write a benchmark program under release mode?

Question

while running some benchmark programs, the wall clock may surprisingly give a very small duration, which is because of various compiler optimization like dead code elimination, loop unwinding... that optimize away the tested code.

I may add some "exterior depency" via static/volatile qualifier, but that's not always work.

Any idea?

Answer 1

Generally, adding "some "exterior depency" via static/volatile qualifier" will change the actual behavior, and timing so is not advised.

The way I generally make sure that code can be tested is by using argc (the first arg to main) If I need a loop, I change the input data using argc . Also have some dummy calculated value that is dependant on the code and printed after the loops and timer to make sure the result can't be optimized out.

If your fill function can take a seed, you can also just use argc as part of the seed.

So if you have random test data, or a loop used to test your code, add argc to it.

Without knowing exactly what you are testing the best I can do is someting like

seed=argc;        // the compiler cannot count on any value of argc
dummy=0;
total_time=0;
for (rep=0; rep < max_rep; ++rep)
{
    for (i =0; i < max_array; ++i)
    {
         test_array[i]=seed+gen_random();  // Note use of seed
    }

    start=time_now();
    dummy+=function_to_time(test_array);  // this is what you are testing
    end=time_now();
    total_time+=end-start;
    seed++;                               // change seed just to be extra paranoid
}
std::cout << "Time=" << total_time << "  Dummy=" << dummy << std::endl;