Latency between thread test: Linux bare metal machine vs QNX on a VM

Question

I recently did a latency comparison on these two setups: a) Ubuntu 16.04 running on a 12 core host; b) A guest QNX running a VMware on a laptop host (4 cores assigned to the QNX VM) - I do not have a better setup currently for QNX.

The test scenario: 10 threads running, each thread sends out a message to a randomly chosen receiving thread every 30ms-ish - very low message rate indeed; the message mechanism is implemented using conditional variables and each thread has its own dedicated rx prod-consumer queue and its own conditional variable and mutex - so no interference between queues. I measure the time between the message got constructed/sent and the receiving thread gets the message. The mean and std_dev min max are all captured for each thread.

The result is surprisingly favoring QNX (although it is running on a VM). 10us vs 40us.

for a thread on Linux (seconds): mean=0.000038076 std_dev=2.7523e-05 min=0.000000254 max=0.000177410 sampleSize=1023 for a thread QNX (seconds): mean=0.000011351 std_dev=0.000105937 min=0.000000000 max=0.000999987 sampleSize=969

I noticed that the QNX side the clock is not as precise (resolution wise) as the Linux side since I do see the min latency measurable is 0.

I just wonder if it conforms with other people's experiences - does Linux thread conditional wake-up take 40us in average? Btw, if the QNX time precision is in 100us and Linux is in nanosec, does this diff impact stat? Thanks.

Answer 1

The latency test turns out to be more sensitive (in a different way) to the HW than what I previous thought. The newer 12 core CPU host has hyper thread enabled and it produces 40us latency numbers. I moved to a older 4 CPU with hyper thread disabled machine, the latency number drops to 15-16us. That is in the same ball park range as the QNX VM number. I wish I could make the platform more aligned in the tests to get a more conclusive answer in the future.

new Linux result: mean=0.000015819 std_dev=1.39205e-05 min=0.000000503 max=0.000117652 sampleSize=1528