What to do when asio::async_write doesn't call the handler?

Question

I have the following two member functions in a class. The _mtxWrite is a mutex object I use to make the write function thread safe. During heavy load, sometimes the writeHandler doesn't get called. Then _mtxWrite doesn't get released, resulting in a deadlock. What is the best way detect the situation and resolve the deadlock?

template <class TSession>
void write(boost::shared_ptr<TSession> pSession, 
    boost::shared_ptr<CInProtocolBase> pMessage)
{
    std::vector<unsigned char>* pData = new std::vector<unsigned char>;
    pMessage->serialize(*pData); 

    _mtxWrite.lock();
    boost::asio::async_write(_socket,boost::asio::buffer(&pData->at(0), 
        pData->size()),
        boost::bind(&this_type::writeHandler<TSession>,
        shared_from_this(),pSession,pData,boost::asio::placeholders::error));
}

template <class TSession>
void writeHandler(boost::shared_ptr<TSession> pSession, 
    std::vector<unsigned char>* pData,const boost::system::error_code& ec)
{
    delete pData;
    _mtxWrite.unlock();

    if(ec)
    {
        _socket.get_io_service().post(boost::bind(&TSession::errorHandler,
            pSession,ec));
    }
}

Answer 1

Your code is locking a mutex where async_write() is called and unlocking it in the handler. If these operations happen in different threads then this will violate the unlock() preconditions (the current thread owns the mutex). This may be causing your deadlock. In general, separated locking and unlocking is usually an indication of suboptimal design.

A better way of handling asynchronous writes would be to use a queue. The logic would look something like:

Write function:

• Lock mutex.
• Push buffer onto queue.
• If queue contains exactly one element, call async_write on the front element.
• Unlock mutex (ideally via a scoped_lock falling out of scope).

Write handler:

• Lock mutex.
• Pop buffer from queue (and free it).
• If the queue is not empty, call async_write on the front element.
• Unlock mutex (ideally via a scoped_lock falling out of scope).

With this approach locking is localized and there is no opportunity for deadlock. @TannerSansbury's pointer in the comments to using a strand for synchronization instead of a mutex will work similarly - it implicitly provides exclusion for any function run in the strand.

I've never had a case of a write handler not being called that wasn't eventually determined to be my own bug, but you can use a boost::asio::deadline_timer to watch for this by setting the timer when you call async_write() and canceling it in the write handler.