Empty http response body

Question

I am trying to make HTTP request/response using sockets in C#. GET request appears below.

 StringBuilder sb = new StringBuilder();
 sb.AppendFormat("GET http://{0}/ HTTP/1.1\r\n", hostname);
 sb.AppendFormat("Host: {0}\r\n", hostname);
 sb.Append("Connection: keep-alive\r\n");
 sb.Append(@"Accept:text/html,*/*");
 sb.Append("\r\n\r\n");

where hostname is something like 'mit.edu' or 'facebook.com' or 'google.com' or anything else. For some strange reason I have just a status-line (with 200 status code) and headers as http response. But there is no message body in the response: attached srceenshot of my console app

Here is a method that perform all manipulations with socket and make http request:

public static void DoHttpGetRequest(String hostname, Int16 port = 80) {
      IPHostEntry ipHostEntry = Dns.GetHostEntry(hostname);
      IPAddress ipAdress = ipHostEntry.AddressList[0];
      IPEndPoint ipEndPoint = new IPEndPoint(ipAdress, port);

      Socket socket = new Socket(ipAdress.AddressFamily, SocketType.Stream, ProtocolType.Tcp);

      socket.Connect(ipEndPoint);

      String request = CreateRequest(hostname);

      Byte[] byteRequest = Encoding.UTF8.GetBytes(request);
      Byte[] byteResponse = new Byte[1000];

      int bytesSent = socket.Send(byteRequest);
      int bytesReceive = socket.Receive(byteResponse);

      Console.WriteLine(request);
      Console.WriteLine();
      Console.WriteLine(Encoding.UTF8.GetString(byteResponse, 0, bytesReceive));

      socket.Shutdown(SocketShutdown.Both);
      socket.Close();
}

My first thought was that the socket hadn't received the whole response from the server. In this case I do still not know how to solve the problem.

So what is going on? Where is the mistake?

Answer 1

Don't expect to get the full response inside a single receive. What you do is to receive data until you have the full response header (this could take several receive calls too), then parse the header to find out how long the response is and then read the necessary data of the response which also can need multiple receive calls. And since you are doing a HTTP/1.1 request you also have to deal with chunked responses.

I recommend to better use a HTTP library to handle all the problems. If you insist on doing it all by your own read the specification of HTTP and implemented accordingly. It also helps to look around at stackoverflow for similar requests because this problem you have is very typical for someone trying to implement HTTP first time, without understanding enough on how sockets, TCP and HTTP work.

Answer 2

I think you should keep the socket open for a longer time. Insert a

Thread.Wait(5000);

may help. Then you can do a second socket.Receive as Steffen proposed. Maybe you can try the following (wait until the server closed the connection):

while (socket.Connected) do
{
    int bytesReceive = socket.Receive(byteResponse);
}

But I didn't test this. It's easier to use the System.Net.Http.HttpClient

Answer 3

Everything I need to do is to receive data with multiple receive calls until all response information received.

do
{                
     bytesReceive = socket.Receive(byteResponse, byteResponse.Length, 0);
     response += Encoding.UTF8.GetString(byteResponse, 0, bytesReceive);
}
while (bytesReceive > 0);

Thanx everybody for your help!