posix线程和O3优化

Question

I am working on a program that uses mpi (openmpi 1.4.3) and pthreads, working in c++ under linux.

some of the mpi nodes have a queuing system implemented with pthreads. Idea is simple one thread adding elements into queue, and few other "working" threads picking up objects and doing their job on them (not a rocket science).

Please consider 2 examples of my working threads which picking up elements. First example working fine unless -O3 optimization is specified. In that case it starts to endlessly looping without picking up anything.

    while (true){
        if (t_exitSignal[tID]){
            dorun = false;
            break;
        }

        //cout<<"w8\n";

        //check if queue has some work for us
        if (!frame_queue->empty()){

            //try to get lock and recheck that queue no empty
            pthread_mutex_lock( &mutex_frame_queue );

            if (!frame_queue->empty()){
                cout<<"Pickup "<<tID<<endl;
                con = frame_queue->front();
                frame_queue->pop();
                t_idling[tID] = false;
                pthread_mutex_unlock( &mutex_frame_queue );
                break;
            }

            pthread_mutex_unlock( &mutex_frame_queue );
        }

    }

Now consider this one, exactly the same code, except mutex gettimg locked before I checking for queue->empthy. This work works fine for all levels of optimization.

    while (true){
        if (t_exitSignal[tID]){
            dorun = false;
            break;
        }
        //cout<<"w8\n";

        //try to get lock and recheck that queue no empty
        pthread_mutex_lock( &mutex_frame_queue );

        //check if queue has some work for us
        if (!frame_queue->empty()){

                cout<<"Pickup "<<tID<<endl;
                con = frame_queue->front();
                frame_queue->pop();
                t_idling[tID] = false;
                pthread_mutex_unlock( &mutex_frame_queue );
                break;

        }
        pthread_mutex_unlock( &mutex_frame_queue );

    }

Just in case it make a difference this is how I populate queue from other thread

                    pthread_mutex_lock( &mutex_frame_queue );
            //adding the same contianer into queue to make it available for threads
            frame_queue->push(*cursor);
            pthread_mutex_unlock( &mutex_frame_queue );

My question is: why first example of code stop working why I compiling with -O3 option ? Any other suggestion for the queuing system ?

Thanks a lot!

SOLUTION: This is what I come up with at the end. Seems to work much better than either of the methods above. (just in case someone interested ;)

    while (true){

        if (t_exitSignal[tID]){

            dorun = false;
            break;
        }
        //try to get lock and check that queue no empty
        pthread_mutex_lock( &mutex_frame_queue );

        if (!frame_queue->empty()){

            con = frame_queue->front();
            frame_queue->pop();
            t_idling[tID] = false;
            pthread_mutex_unlock( &mutex_frame_queue );
            break;
        }else{

            pthread_cond_wait(&conf_frame_queue, &mutex_frame_queue);
            pthread_mutex_unlock( &mutex_frame_queue );
        }




    }

Adding

        pthread_mutex_lock( &mutex_frame_queue );

        //adding the same contianer into queue to make it available for threads
        frame_queue->push(*cursor);
        //wake up any waiting threads
        pthread_cond_signal(&conf_frame_queue);
        pthread_mutex_unlock( &mutex_frame_queue )

Answer 1

I'm tempted to suggest __sync_synchronize() before the first empty check, but that's probably not safe—if another thread's in the middle of adding to the container, that container may still be in an inconsistent state when you call empty() . Depending on the container, anything could happen, from getting back the wrong answer to crashing.

Josh is probably right too. Locking a mutex also provides a memory barrier, which among other things means your code will re-read the memory it's using to determine whether the container's empty each time. Without some sort of memory barrier, that's never actually guaranteed to happen, so at higher optimization levels your code may never see the change.

Also, have you looked into pthread's condition variables ? They would allow you to avoid polling in a loop until your container's no longer empty.

Answer 2

我想您会在检查队列是否为空时看到基于指令排序假设的错误-当您启动优化时，顺序会更改，并且由于您没有设置互斥锁而无法保护内存，该顺序会中断这是发生的。