How to buffer data for send() and select()?

Question

While a send() succeeds with all data being sent most of the time, it is not always the case. Thus people are advised to use the write-fdset for select() and poll() to check when the socket is writeable.

How do usual mechanisms look like to actually buffer the data to send while still maintaining a well comprehensible sourcecode?

Answer 1

I'm not deeply familiar with the *nix side of socket programming, but I encountered the same problem on Win32 side. Buffering is not so much of a problem (you queue the requests and peek on write completion to submit next from queue), the real problem is that the need to buffer signals that you're actually handling flow control and you cannot resolve flow-control with buffering alone: there can always be a consumer slower than the producer and the buffer will basically grow out of control. You have to propagate the flow control up the stream to whatever module is producing the data, and this makes really complex interfaces. All 'write' requests must support return codes indicating flow control status (ie. 'stop writing, there is no more space!') and callbacks to invite the caller to resume write operations.

Answer 2

As we're in C++ land, you could store the data in a std::vector

New data gets appended to the end of the vector. When you get notification that the socket is writable, try to send the complete vector. send() will return how much was really sent. Then simply erase that number of bytes from the beginning of the vector:

std::vector<char> buffer;
...
if( ! buffer.empty() )
{
    int bytesRead = send( socket, &buffer[ 0 ], buffer.size(), flags );
    if( bytesRead > 0 )
        buffer.erase( 0, bytesRead );
    else
       // some error...
}

So there's probably more error checking to do, but you get the idea?

Rather queueing each individual send request, the advantage here is that you get to potentially combine multiple higher level sends into one socket send, assuming you're using TCP?

But as Remus quite rightly mentions, your flow control and API is the tricky bit - ie how do you stop the buffer becoming too big?

Answer 3

When writing OO that needs to center around a select()/poll() loop, you need a good abstraction. I have always found the Adaptive Communications Environment (ACE) Reactor class very good for this. There are a couple of books "C++ Network Programming" by Doug Schmidt that cover this environment also various other stuff on the web including The Design and Use of the ACE Reactor