Behavior of writing to TCP socket before reading all data

Question

I've been writing small specific purpose HTTP servers for some applications of mine, and I noticed that, if you write() before you read() all available data, the bytes are not sent properly. For example, after read() ing only the request line ( GET / HTTP/1.1\\r\\n ) sent by my browser, I write() :

HTTP/1.1 200 OK\r\n
Connection: close\r\r
Content-Type: text/html\r\n
\r\n
(some HTML stuff)

Wireshark capture of this write() :

'\\n' bytes and Content-Type header are gone! (Wireshark always displays '\\n' bytes in HTTP header section, if they exist)

And the browser do not displays the HTML content.

So I should never write() before read() everything? Is this the TCP standard?

Edit 1: adding my C++ code that sends the stuff:

string header =
  "HTTP/1.1 200 OK\r\n"
  "Connection: close\r\r"
  "Content-Type: text/html\r\n"
  "\r\n"
;
write(sd, header.c_str(), header.size()); // from unistd.h
FILE* fp = fopen("index.html", "rb");
char by;
while (fread(&by,1,1,fp) == 1) write(sd,&by,1);
fclose(fp);

Edit 2: Well, @selbie pointed a typo... "Connection: close\\r\\r" . After fixing it, the behavior has changed, to a less scary one: the write() is actually not sending data at all. Now, Wireshark only displays the request! No response (from my write() ) is captured.

Edit 3: As suggested by @usr, I wrote a small test client... When the server read() s everything before write() ing, the client always receives all the HTTP payload. When the server write() s before read() ing the headers sent by the client, the client never receives the whole HTTP payload. I did many tests!

When the server write() s after read() ing the headers:

HTTP/1.1 200 OK\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
<form>\n
  <input type="text" name="field1" />\n
  <input type="text" name="field2" />\n
  <input type="submit" value="send" />\n
</form>\n

When the server write() s before read() ing the headers:

HTTP/1.1 200 OK\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
<form>\n
  <input type="text" name="field1" />\n
  <input

and:

HTTP/1.1 200 OK\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
<form>\n
  <input type="text" name="field1" />\n
  <input type="text" name="field2" />\n
  <input type="submit"

and:

HTTP/1.1 200 OK\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
<form>\n
  <input type="tex

I ran the client like 50 times with each setting.

Why is this happening??? It has to be something with the kernel...

Edit 4: Another thing I noticed doing these tests... Wireshak always displays both request and response, if the server read() s the headers, but always displays only the request, if the server do not read() the headers. Seriously, this has to do with TCP.

Answer 1

You should not write the response before reading the request. You are violating the HTTP protocol.

That said I don't know why the browser would behave like that. In any case stop violating the HTTP protocol.

TCP is a bidirectional stream of bytes. It does not care when and what is written. This is not a TCP-level issue.

I'm not sure what I'm seeing on that screenshot. If you mean the missing \\n chars that certainly was not stripped by the kernel. The kernel has no business interfering with the data you are sending. It does not know what the data means.

Your app has a bug. Maybe you are using some library that "helpfully" converts the line endings to Linux format?! Impossible to answer without code. This answer is as good as it gets with the information posted.

Answer 2

Your Connection header ends with \\r\\r instead of \\r\\n That explains the weirdness in your Wireshark trace.

Instead of this;

"Connection: close\r\r"

Change it do this:

"Connection: close\r\n"

Answer 3

Well... Seems like the kernel has the following policy, which I discovered only by empirical testing, putting a sleep(1) before close() .

If there is nothing left to be read() , and you immediately call write() and close() , the kernel will send everything properly, no problem.

But if there is stuff to be read() and you write(); close() write(); close() , the kernel will stop sending the data, like if you just decided to stop the conversation abruptly. The kernel kinda thinks "well... he didn't even read() this stuff I'm holding and he's closing the socket... he probably also wants me to stop sending this stuff he told me to send". Stupid kernel!