更好的基准同步

Question

I'm trying to benchmark some interrupt functionality that I added to a kernel. For the time being, I just want to measure how long it takes for an interrupt to be sent from one core and received on another. I'm roughly doing the following:

volatile bool wait = true;

...

//Sending core:
void run_benchmark() {
    //clear pipeline and record time A with rdtsc
    for (int i = 0; i < 10000; i++) {
        send_interrupt();
        while (wait);
        wait = true;
    }
    //record time B with rdtsc
    //benchmark = (B - A) / 10000
}

...

//Receiving core:
void handle_interrupt(...) {
    wait = false;
    ...
}

I also subtract other overheads out of the benchmark, such as the cost of recording a time, etc. I send the interrupt 10,000 times in order to get a stable value.

My main concern with this approach is that there will be a cache miss on both the receiving core and the sending core, since they each set wait to a different value. Given how fast interrupt delivery already is, these cache misses are likely having a significant effect on my recorded benchmark.

Is there a better way to do this?

Answer 1

On newer Intel platforms the TSC's of all cores should be synchronized under Linux. So I don't think you need this kind of synchronization (see corresponding thread in the Intel developer zone).

Why dont you simply take the TSC value on the receiving CPU? Then you can wait on the sending CPU until the variable for time B contains a plausible value.

Answer 2

Yes, both L1 caches on each core will be flapping because each one is invalidating each other's cache lines.

I do not know how did you implemented the send_interrupt() but if it is by using some shared variable, then you also need to consider that there is cache evictions for this communication, which incurs in some delays on measured values.

About measuring the execution time,I was working recently this stuff, here is the code I used to measure clock ticks:

uint64_t rdtscp(uint64_t *chip, uint64_t *core)
{
    uint32_t a, d, c;

    __asm__ volatile("rdtscp" : "=a" (a),"=d" (d), "=c" (c));
    *chip = (c & 0xFFF000) >> 12;
    *core = c & 0xFFF;

    return ((uint64_t) a) | (((uint64_t) d) << 32);
}

Basically the RDTSCP is synchronized over all the cores that share the same chipset in a NUMA architecture, and the rdtscp instruction uses three different CPU registers to form the 64-bits timestamp, that is why it requires those three variables in the function.

Nonetheless, you can get other useful information from this instruction, such as the core id and chip id, for instance.

So, you can use this timestamp to measure the time your code takes.

Other thing that may be useful is to pin your process to a specific CPU-core, because it may be scheduled to a different core, so it also incurs in task migration and so on.

PS: its resolution is in CPU clock ticks.