C++ Blocking Queue Segfault w/ Boost

Question

I had a need for a Blocking Queue in C++ with timeout-capable offer() . The queue is intended for multiple producers, one consumer . Back when I was implementing, I didn't find any good existing queues that fit this need, so I coded it myself.

I'm seeing segfaults come out of the take() method on the queue, but they are intermittent. I've been looking over the code for issues but I'm not seeing anything that looks problematic.

I'm wondering if:

• There is an existing library that does this reliably that I should use (boost or header-only preferred).
• Anyone sees any obvious flaw in my code that I need to fix.

Here is the header:

class BlockingQueue
{
    public:
        BlockingQueue(unsigned int capacity) : capacity(capacity) { };
        bool offer(const MyType & myType, unsigned int timeoutMillis);
        MyType take();
        void put(const MyType & myType);
        unsigned int getCapacity();
        unsigned int getCount();

    private:
         std::deque<MyType> queue;
         unsigned int capacity;
};

And the relevant implementations:

boost::condition_variable cond;
boost::mutex mut;

bool BlockingQueue::offer(const MyType & myType, unsigned int timeoutMillis)
{
    Timer timer;

    // boost::unique_lock is a scoped lock - its destructor will call unlock().
    // So no need for us to make that call here.
    boost::unique_lock<boost::mutex> lock(mut);

    // We use a while loop here because the monitor may have woken up because
    // another producer did a PulseAll. In that case, the queue may not have
    // room, so we need to re-check and re-wait if that is the case.
    // We use an external stopwatch to stop the madness if we have taken too long.
    while (queue.size() >= this->capacity)
    {
        int monitorTimeout = timeoutMillis - ((unsigned int) timer.getElapsedMilliSeconds());

        if (monitorTimeout <= 0)
        {
            return false;
        }

        if (!cond.timed_wait(lock, boost::posix_time::milliseconds(timeoutMillis)))
        {
            return false;
        }
    }

    cond.notify_all();

    queue.push_back(myType);

    return true;
}

void BlockingQueue::put(const MyType & myType)
{
    // boost::unique_lock is a scoped lock - its destructor will call unlock().
    // So no need for us to make that call here.
    boost::unique_lock<boost::mutex> lock(mut);

    // We use a while loop here because the monitor may have woken up because
    // another producer did a PulseAll. In that case, the queue may not have
    // room, so we need to re-check and re-wait if that is the case.
    // We use an external stopwatch to stop the madness if we have taken too long.
    while (queue.size() >= this->capacity)
    {
        cond.wait(lock);
    }

    cond.notify_all();

    queue.push_back(myType);
}

MyType BlockingQueue::take()
{
    // boost::unique_lock is a scoped lock - its destructor will call unlock().
    // So no need for us to make that call here.
    boost::unique_lock<boost::mutex> lock(mut);

    while (queue.size() == 0)
    {
        cond.wait(lock);
    }

    cond.notify_one();

    MyType myType = this->queue.front();

    this->queue.pop_front();

    return myType;
}

unsigned int BlockingQueue::getCapacity()
{
    return this->capacity;
}

unsigned int BlockingQueue::getCount()
{
    return this->queue.size();
}

And yes, I didn't implement the class using templates - that is next on the list :)

Any help is greatly appreciated. Threading issues can be really hard to pin down.

-Ben

Answer 1

Why are cond, and mut globals? I would expect them to be members of your BlockingQueue object. I don't know what else is touching those things, but there may be an issue there.

I too have implemented a ThreadSafeQueue as part of a larger project:

https://github.com/cdesjardins/QueuePtr/blob/master/include/ThreadSafeQueue.h

It is a similar concept to yours, except the enqueue (aka offer) functions are non-blocking because there is basically no max capacity. To enforce a capacity I typically have a pool with N buffers added at system init time, and a Queue for message passing at run time, this also eliminates the need for memory allocation at run time which I consider to be a good thing (I typically work on embedded applications).

The only difference between a pool, and a queue is that a pool gets a bunch of buffers enqueued at system init time. So you have something like this:

ThreadSafeQueue<BufferDataType*> pool;
ThreadSafeQueue<BufferDataType*> queue;

void init()
{
    for (int i = 0; i < NUM_BUFS; i++)
    {
        pool.enqueue(new BufferDataType);
    }
}

Then when you want send a message you do something like the following:

void producerA()
{
    BufferDataType *buf;
    if (pool.waitDequeue(buf, timeout) == true)
    {
        initBufWithMyData(buf);
        queue.enqueue(buf);
    }
}

This way the enqueue function is quick and easy, but if the pool is empty, then you will block until someone puts a buffer back into the pool. The idea being that some other thread will be blocking on the queue and will return buffers to the pool when they have been processed as follows:

void consumer()
{
    BufferDataType *buf;
    if (queue.waitDequeue(buf, timeout) == true)
    {
        processBufferData(buf);
        pool.enqueue(buf);
    }
}

Anyways take a look at it, maybe it will help.

Answer 2

I suppose the problem in your code is modifying the deque by several threads. Look:

1. you're waiting for codition from another thread;
2. and then immediately sending a signal to other threads that deque is unlocked just before you want to modify it;
3. then you modifying the deque while other threads are thinking deque is allready unlocked and starting doing the same.

So, try to place all the cond.notify_*() after modifying the deque. Ie:

void BlockingQueue::put(const MyType & myType)
{
    boost::unique_lock<boost::mutex> lock(mut);
    while (queue.size() >= this->capacity)
    {
        cond.wait(lock);
    }

    queue.push_back(myType);  // <- modify first

    cond.notify_all();        // <- then say to others that deque is free
}

For better understanding I suggest to read about the pthread_cond_wait() .