Linux上共享内存写入的周期性延迟峰值

Question

I have the following code:

#pragma pack(4)
struct RECORD_HEADER {
uint64_t msgType;
uint64_t rdtsc;
};
struct BODY {
    char content[488];
};
#pragma pack()

class SerializedRDTSC {
public:
    typedef unsigned long long timeunit_t;

    static timeunit_t start(void) {
            unsigned cycles_high, cycles_low;
            __asm__ __volatile__ (  "CPUID\n\t"
                                    "RDTSC\n\t"
                                    "mov %%edx, %0\n\t"
                                    "mov %%eax, %1\n\t": "=r" (cycles_high), "=r" (cycles_low)::
                                    "%rax", "%rbx", "%rcx", "%rdx");
            return ( (unsigned long long)cycles_low)|( ((unsigned long long)cycles_high)<<32 );
    }

    static timeunit_t end(void) {
            unsigned cycles_high, cycles_low;
            __asm__ __volatile__(   "RDTSCP\n\t"
                                    "mov %%edx, %0\n\t"
                                    "mov %%eax, %1\n\t"
                                    "CPUID\n\t": "=r" (cycles_high), "=r" (cycles_low):: "%rax",
                                    "%rbx", "%rcx", "%rdx");
            return ( (unsigned long long)cycles_low)|( ((unsigned long long)cycles_high)<<32 );
    }

};

char* createSHM() noexcept {
        const auto sharedMemHandle = shm_open("testing", O_RDWR | O_CREAT, 0666);
        if (-1 == sharedMemHandle) {
            std::cout << "failed to open named shared memory: " << std::endl;
            return nullptr;
        }
        constexpr int32_t size = (1 << 26);
        ftruncate(sharedMemHandle, size);
        char* ptr = (char*) mmap(nullptr, size, PROT_READ | PROT_WRITE,
                MAP_SHARED, sharedMemHandle, 0);

        if (MAP_FAILED == ptr) {
            std::cout << errno << std::endl;
            return nullptr;
        }

        const auto rc = fchmod(sharedMemHandle, 0666);
        if (rc == -1) {
            fprintf(stderr,
                    "Can't change permissions to 0666 on shared mem segment: %m\n");
            fflush(stderr);
        }
        return ptr;
}

int main() {
    BODY update;

    srand(time(nullptr));
    char* ptr = createSHM();

    constexpr uint64_t n = 700;
    constexpr uint64_t n2 = 10;
    uint64_t m_data[n * n2];
    memset(m_data, 0, sizeof(m_data));

    uint64_t r = 0;

    for (uint64_t i = 0; i < n; i++) {
        for (uint64_t k = 0; k < n2; k++) {
            // populate the header
            const auto msgType = rand();
            const auto rdtsc = rand();

            // populate the struct randomly
            uint32_t* tmp = reinterpret_cast<uint32_t*>(&update);
            for (uint32_t j = 0; j < sizeof(BODY) / sizeof(uint32_t); j++) {
                const uint32_t v = rand() % 32767;
                tmp[j] = v;
            }

            // write the struct
            const auto s = SerializedRDTSC::start();
            memcpy(ptr, (char*)&msgType, sizeof(uint64_t));
            ptr+= sizeof(uint64_t);
            memcpy(ptr, (char*)&rdtsc, sizeof(uint64_t));
            ptr+= sizeof(uint64_t);
            memcpy(ptr, &update, sizeof(BODY));
            ptr+= sizeof(BODY);
            const auto e = SerializedRDTSC::end();
            m_data[r++] = e - s;
        }
        usleep(249998);
    }

    for (uint32_t i = 0; i < r; i++) {
        std::cout << i << "," << m_data[i] << std::endl;
    }
}

And for some reason, there are periodic latency spike according to the output:

0   9408
1   210
2   162
3   176
4   172
5   164
6   172
7   8338
8   174
9   186
10  612
11  380
12  380
13  374
14  358
15  13610
16  190
17  186
18  164
19  168
20  246
21  196
22  170
23  5066
24  176
25  176
26  168
27  174
28  166
29  440
30  232
31  214
32  5128
33  180
34  178
35  172
36  174
37  184
38  170
39  162
40  5964
41  182
42  174
43  164
44  180
45  180
46  162
47  172

I already isolated the core and double-checked with htop to make sure no other processes were using the core.

My machine has an i7 CPU (nothing fancy).

And then I tried with an Xeon CPU. The pattern is about the same -- every 7-11 write, there was a spike.

With i7 CPU, I compiled with GCC 7.2 with c++17 and ran it on CentOS 7.3.

With Xeon CPU, I compiled with GCC 4.6 with c++0x and ran it on CentOS 6.5.

My questions are: 1. Why there were periodic latency spikes? (I checked with strace. And I don't see weird system call involved) 2. Any suggestion on how to investigate/understand the spike? More for my learning.

Thanks in advance!

PS Yes, some people object to use rdtsc to measure latency because temperature affects TSC. Tho, I don't see any better option as I don't have PTP, and clock_gettime() sometimes will have latency spikes too. If you have any suggestion, it is more than welcome :)

Answer 1

A memory page is 4K bytes. Every time you start writing on a new page, that page needs mapped into the process address space. Since the data you're writing every loop is 8 + 8 + 488 = 504 bytes, you'll get a spike every 8 or 9 time thru the loop.

Since the CPU can speculatively prefetch data from memory, the page fault for the 2nd page (which should occur on the 8th loop) occurs one loop earlier than expected, when the hardware prefetcher tries to access the page.