System.Net.WebResponse throwing “System.Net.WebException: The operation has timed out” but connection to web server was made

Question

I have a method which is intended to download a file from an HTTP URL to a byte array:

private static byte[] DownloadFileToByteArrayWorker(HttpWebRequest Request, int bufferLength)
{
    byte[] responseByes = null;

    //Round up to the nearest multiple of 1024
    bufferLength = AdjustBufferLength(bufferLength);

    Request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
    Request.ServicePoint.Expect100Continue = false;
    Request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");

    using (var Response = (HttpWebResponse)Request.GetResponse())
    {
        using (Stream ResponseStream = Response.GetResponseStream())
        {
            using (MemoryStream ms = new MemoryStream())
            {
                int count = 0;
                byte[] buf = new byte[bufferLength];
                while ((count = ResponseStream.Read(buf, 0, buf.Length)) > 0)
                {
                    ms.Write(buf, 0, count);
                }
                responseByes = ms.ToArray();
            }
        }
    }

    return responseByes;
}

Request.GetResponse() is throwing a time out exception no matter how long I make the Timeout property of the HttpWebRequest. I can verify via my logs that the program is waiting the full Timeout period before erroring out, however, correlating my logs with the web server logs indicates that the web server is sending back a response almost immediately.

An interesting note is that when I access the same web server via the load balancer rather than directly, it downloads the file practically instantly. Also, if I access the URL via the web server directly in a web browser (no proxy needed, btw) I can download the file from individual web servers instantly that way too.

Some additional details:

• I am using .NET Framework 4.7 on Windows 2012 R2.
• The web server I'm trying to connect to is Apache on RHEL7. I'm not sure about the specific Apache version
• I am connecting to the web server on a specific port which is reserved for HTTP traffic (a separate website is hosted on a different port number for HTTPS)
• There's no web proxy

Any suggestions?

Answer 1

As you said your code has problem only when you call the load balancer, I think the problem is the your client send a 100 continue request but your load balancer don't know how to handle it.

That is the reason you client doesn't send all the data right after the beginning of connection.

You can find more information about 100 continue in HTTP rfc section 8.2.3.

To fix the behavior from client side in c# you have to add this code:

ServicePointManager.Expect100Continue = false;

You can see the full documentation about this feature here .