Reasons for a slow recv call

Question

I'm writing two applications (in C), which make several calls to send and receive (eg I'm implementing a remote file copy).

I always send a 64 Byte Header which contains the length of the following message body and some other information.

When testing my application on some files I recognized that some recv calls take a pretty long time to complete (about 40 ms ). Using strace I found out that it happens first when sending a message body of 377 Bytes (which in this case it the whole content of my file to copy).

The server application starts sending the message body which takes about 48 us . Now the client application consumes about 38 ms to receive these bytes.

From that time on every receive calls consumes that much time, since they are each blocking in a receive and waiting for the reply.

server's strace

[pid 27158] 1292236124.465827 send(6, "\\0\\0\\1\\271\\0\\0\\0\\0\\0\\0\\0\\0core.fwrite\\0\\0\\0\\0\\0\\0\\0\\0\\0"..., 64, 0) = 64 <0.000031>

[pid 27158] 1292236124.466074 send(6, "\\0\\0\\0\\1\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\10\\0\\0\\0\\1\\0\\0\\0\\0\\0\\0\\0\\1\\0\\0\\0\\0"..., 377, 0) = 377 <0.000048>

client's strace

[pid 27159] 1292236124.466364 recv(4, "\\0\\0\\1\\271\\0\\0\\0\\0\\0\\0\\0\\0core.fwrite\\0\\0\\0\\0\\0\\0\\0\\0\\0"..., 64, 0) = 64 <0.000027>

[pid 27159] 1292236124.466597 recv(4, "\\0\\0\\0\\1\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\10\\0\\0\\0\\1\\0\\0\\0\\0\\0\\0\\0\\1\\0\\0\\0\\0"..., 377, 0) = 377 <0.037456>

This problem really gives me a hard time since I do not understand why the receive call on the client takes so much time.

Any hints would be highly appreciated.

Answer 1

Sounds like Nagle's algorithm to me. You're not submitting enough data, so it delays a while in case there's more coming. You can disable it via a socket option and try again.

Answer 2

Yes, that's definitely Nagle's algorithm kicking in. I suggest you read up on it since if you are sending large chunks of data it should send as soon as it has "lots" of data. Without reading up myself I'm not entirely sure how much "lots" is, but it's probably configurable.

However if you are doing one file per connection, then you shouldn't have a problem with small files if you send the header and the content in one go - which you should be doing anyway to improve throughput. As Lijie says (more or less) "throughput is the name of the game". You need to be buffering your header together with the file data. This is one of the few situations where I might recommend multi-threading - one thread to fill the buffer from the file and another to load the socket from the buffer. You'll need to protect the buffer and associated variables with a mutex and I recommend using a condition variable too. The file reading thread signals when it has added more data and waits when the buffer is full, and the socket writing thread signals when it has read and waits when the buffer is empty. The file reading thread should not signal after writing the 64 byte header. Let it load a full buffer of data first.

You could also try having two buffers and alternate using them to reduce mutex lock delays. If you get it right, the file reading thread will be writing to buffer A while the socket writing thread is reading buffer B and visa versa and the threads will be waiting on a mutex less often.

Even with such a strategy it might still be worth disabling Nagle's algorithm. Apart form anything, if your code is desinged to avoid lots of small packets anyway, then Nagle's algorithm is redundant.