在单个内核上运行的多个线程如何进行数据竞争？

Question

I have the following simple c++ source :

#define CNTNUM 100000000
int iglbcnt = 0 ;
int iThreadDone = 0 ;

void *thread1(void *param)
{
    /*
    pid_t tid = syscall(SYS_gettid);
    cpu_set_t set;
    CPU_ZERO( &set );
    CPU_SET( 5, &set );
    if (sched_setaffinity( tid, sizeof( cpu_set_t ), &set ))
    {
        printf( "sched_setaffinity error" );
    }
    */
    pthread_detach(pthread_self());
    for(int idx=0;idx<CNTNUM;idx++)
        iglbcnt++ ;
    printf(" thread1 out \n") ;
    __sync_add_and_fetch(&iThreadDone,1) ;
}

int main(int argc, char **argv)
{
    pthread_t tid ;
    pthread_create(&tid , NULL, thread1, (void*)(long)1);
    pthread_create(&tid , NULL, thread1, (void*)(long)3);
    pthread_create(&tid , NULL, thread1, (void*)(long)5);
    while( 1 ){
        sleep( 2 ) ;
        if( iThreadDone >= 3 )
            printf("iglbcnt=(%d) \n",iglbcnt) ;
    }
}

If I run it , the answer should not be 300000000 for sure unless the source using __sync_add_and_fetch(iglbcnt, 1 ) instead of iglbcnt++ .

Then I try to run like numactl -C 5 ./x.exe , numactl try to affinity all 3 thread1 to run at core 5 , so in theory , there is only one of all 3 thread1 can be running at core 5 , and since iglbcnt is globar vars to all thread1 , I expect the answer would be 300000000 , unfortunately it is not all the time get 300000000 , sometimes come out like 292065873 .

I guess the reason why not always get 300000000 is that while doing context switch in core 5 , the value of iglbcnt still keep in cpu's store buffer , so when scheduler run another thread then value of iglbcnt in L1 cache would be different with value in cpu core 5's store buffer , that cause the answer comes 292065873 , not 300000000 .

This is only experiment , as I said __sync_add_and_fetch will solve problem, but still I like to know the detail to cause this result .

Edit :

Both ++igblcnt and igblcnt++ produce the same code.

g++ --std=c++11 -S -masm=intel x.cpp ,(source ++iglbcnt) the following code come from xs :

.LFB11:
    .cfi_startproc
    push    rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    mov     rbp, rsp
    .cfi_def_cfa_register 6
    sub     rsp, 32
    mov     QWORD PTR [rbp-24], rdi
    call    pthread_self
    mov     rdi, rax
    call    pthread_detach
    mov     DWORD PTR [rbp-4], 0
    jmp     .L2
.L3:
    mov     eax, DWORD PTR iglbcnt[rip]
    add     eax, 1
    mov     DWORD PTR iglbcnt[rip], eax
    add     DWORD PTR [rbp-4], 1
.L2:
    cmp     DWORD PTR [rbp-4], 99999999
    jle     .L3
    mov     edi, OFFSET FLAT:.LC0
    call    puts
    lock add        DWORD PTR iThreadDone[rip], 1
    leave
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE11:
    .size   _Z7thread1Pv, .-_Z7thread1Pv
    .section        .rodata
.LC1:
    .string "iglbcnt=(%d) \n"
    .text

Edit2 :

for(int idx=0;idx<CNTNUM;idx++){
    asm volatile("":::"memory") ;
    iglbcnt++ ;
}

and then compile it by -O1 will works fine , add compiler-time memory barrier would help in this case .

Answer 1

igblcnt++ is a load, add, store sequence. This is performed without synchronization so threads (even if scheduled on the same core) will have a race because each of them has their own register context. A __sync_add_and_fetch instruction on igblcnt will resolve the race.

The load into a core's register takes place then the thread is switched out (it's registers are saved) another thread reads the same value and increments and stores it back to memory (perhaps hundreds of increments) and then the first thread is switched in with its stale value which is incremented and stored -losing thousands to millions of increments potentially (as you have seen).

Answer 2

Threads running on one processor can have a data race if they are preemptively scheduled, meaning that an interrupt can occur at any moment which triggers a thread context switch. Threads then have to use mutual exclusion mechanisms like mutex objects, or else atomic instructions (together with a carefully designed algorithm).

Cooperatively scheduled threads on a single processor avoid data races implicitly. Under cooperative threading on a single processor, one thread executes until it explicitly calls some function which switches context. Any code which doesn't call such a function is free from interference from other threads.