write to boost::asio socket from different threads

Question

In our application we use Boost libraries (and ASIO for network communications).

Recently, we discovered that if we're sending our data from different threads via same socket, our client application is receiving garbaged data.

Small test to highlight the issue:

#include <stdio.h>
#include <boost/thread.hpp>
#include <boost/asio.hpp>

void send_routine(boost::shared_ptr<boost::asio::ip::tcp::socket> s, char c)
{
  std::vector<char> data(15000, c);
  data.push_back('\n');

  for (int i=0; i<1000; i++)
    boost::asio::write(*s, boost::asio::buffer(&data[0], data.size()));
}


int main()
{
  using namespace boost::asio;
  using namespace boost::asio::ip;

  try {
    io_service io_service;
    io_service::work work(io_service);

    const char* host = "localhost";
    const char* service_name = "18000";

    tcp::resolver resolver(io_service);
    tcp::resolver::query query(tcp::v4(), host, service_name);
    tcp::resolver::iterator iterator = resolver.resolve(query);

    auto socket = boost::shared_ptr<tcp::socket>(new tcp::socket(io_service));
    socket->connect(*iterator);

    boost::thread t1(send_routine, socket, 'A');
    boost::thread t2(send_routine, socket, 'B');
    boost::thread t3(send_routine, socket, 'C');

    t1.join();
    t2.join();
    t3.join();
  }
  catch (std::exception& e) {
    printf("FAIL: %s\n", e.what());
  }
    return 0;
}

So, we create socket here, connect to localhost:18000 and start 3 threads which will write to the socket.

In different terminal window, I run nc -l -p 18000 | tee out.txt | sort | uniq | wc -l nc -l -p 18000 | tee out.txt | sort | uniq | wc -l nc -l -p 18000 | tee out.txt | sort | uniq | wc -l . I expect 3 as output, but it returns more then 100 "different strings" in the network stream (so, data is corrupted). But it works with small buffer sizes (if we'll change 15000 to 80 , for example).

So, the question is: is it a correct behavior of ASIO library? And another: how to fix it? Should I use mutex inside my send_routine function (or there is another solution)?

Answer 1

write and async_write are not thread safe in the manner you are using them. The canonical way to approach this is to queue your messages, then write them out one at a time.

Answer 2

Well according to the documentation tcp::socket is not thread safe when shared between multiple threads.
So you either do a synchronisation like you suggested with boost::mutex or you use async write. The io_service the work for you.

Answer 3

Yes there is another solution ! Strands: Use Threads Without Explicit Locking . Be care that strands only provides "atomic" access to socket for the "event handlers", of course you need to use asio "event handlers" which is not the case of your code. In other words you need to use boost::asio::async_write instead of boost::asio::write.

Answer 4

You might have two problems, the threading issue could be solve for example by having one thread dedicated to writing and a queue where all threads post there response. You can also change your design to an asynchronous one and use the write_some() function and let the threading be done by the io_service::run(), which can be run by more than one thread.

Second, you might have a protocol problem, if a client expects the answers to it's questions in the same order.

hth

Torsten