如何在std :: deque中避免`deque iterator not dereferencable`？ 锁？

Question

Currently in my project I have two static methods PushObjects and ProcessObject . The PushObject method pushes back data to a static deque and this method can be accessed by multiple threads but the ProcessObject is always used by a single thread and is used to retrieve objects from the top and remove them.Now my problem is no matter what I try I always eventually (sooner or later get a deque iterator not dereferencable error.Any suggestions on what I could do to stop this issue. An abstract of my PushObjects and ProcessObject are given below

    void foo::PushObjects(obj A)
    {
        try
        {
            {//Begin Lock
                boost::lock_guard<boost::mutex> lock(mutex_push);
                mydeque.push_back(A);
            }//End Lock
            condition_read.notify_one(); //Inform the reader that it could start reading 
        }
        catch (std::exception& e)
        {
            __debugbreak();
        }
    }


    This is the static Reader method

    void foo::ProcessObject()
    {
        {//Begin Lock
            boost::unique_lock<boost::mutex> lock(mutex_process);
            while(true)
            {
                    while(mydeque.empty()) { condition_read.wait(lock); }
                    try
                    {
                        if(!mydeque.empty())
                        {
                                obj a = mydeque.front();
                                ......Process the object........
                                mydeque.pop_front();
                        }

                    }
                    catch (std::exception& e)
                    {
                        __debugbreak();
                    }
            }//end while
        }//End lock
    }

From what I have read is that iterators become invalid once items are added or removed from the deque. Is there a way to resolve this issue.

Answer 1

It looks like you aren't using the same mutex ( mutex_push vs mutex_process ) for reading and writing from/to the deque . You need to be. It's not safe to write to memory and read from it at the same time on different threads.

Other notes:

obj a = mydeque.front();
......Process the object........
mydeque.pop_front();

could be much better if you manage your lock to have the minimum lock time...

obj a = std::move(mydeque.front());
mydeque.pop_front();
lock.unlock();
// process the object
lock.lock();

You likely don't need to be locked (or at least don't need the same lock) to process the object. This way your writer can still write to the deque while you are processing. Also to note is that there is nothing stopping you here from being multi producer multi consumer instead of just multi producer single consumer.

Answer 2

To expand Dave's answer regarding unlocking as soon as possible to allow concurrent writes while you are processing your items...

Your deque may well contain several items (which have been pushed eg. while you were still processing another item). In order to avoid locking for each and every item you could just swap your deque with an empty, local one and process your items from that local deque. Some code will make it clearer:

while (true) {
    std::deque<Obj> tmp_deque;
    {
        std::unique_lock<std::mutex> lock(mutex);
        while (mydeque.empty())
            condition.wait(lock);
        mydeque.swap(tmp_deque);
    }
    while (!tmp_deque.empty()) {
        Obj obj = std::move(tmp_deque.front());
        tmp_deque.pop_front();
        // process obj
    }
}

This way, instead of lock / get 1 item / unlock / process 1 item you end up with lock / get all items / unlock / process all items which is much more efficient since locking mutexes are a big performance hit.

Clearly, this is only for a single-consumer model . If you have multiple consumers you really don't want to enqueue all items in a single consumer and let all the other consumers go idle.

Answer 3

You need to have single mutex for accessing mydeque no read/write mutexes. And any access to deque must be while mutex locked. Even you just checking empty(). Since deque operations are not atomic you may end up with mydeque.empty() returning false, while being in some seminonempty state in a middle of push_back. Thus you need boost::lock_guard<boost::mutex> lock(mutex_push); prior to each access to the mydeque. or during whole operation that changes deque content.