C# - Is there a limit to the size of an httpWebRequest stream?

Question

I am trying to build an application that downloads a small binary file (20-25 KB) from a custom webserver using httpwebrequests.

This is the server-side code:

Stream UpdateRequest = context.Request.InputStream;
byte[] UpdateContent = new byte[context.Request.ContentLength64];
UpdateRequest.Read(UpdateContent, 0, UpdateContent.Length);
String remoteVersion = "";
for (int i = 0;i < UpdateContent.Length;i++) { //check if update is necessary
    remoteVersion += (char)UpdateContent[i];
}

byte[] UpdateRequestResponse;

if (remoteVersion == remotePluginVersion) {
    UpdateRequestResponse = new byte[1];
    UpdateRequestResponse[0] = 0; //respond with a single byte set to 0 if no update is required
} else {
    FileInfo info = new FileInfo(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
    UpdateRequestResponse = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
    //respond with the updated file otherwise
}

//this byte is past the threshold and will not be the same in the version the client recieves
Console.WriteLine("5000th byte: " + UpdateRequestResponse[5000]);

//send the response
context.Response.ContentLength64 = UpdateRequestResponse.Length;
context.Response.OutputStream.Write(UpdateRequestResponse, 0, UpdateRequestResponse.Length);
context.Response.Close();

After this the array UpdateRequestResponse contains the entire file and has been sent to the client.

The client runs this code:

//create the request
WebRequest request = WebRequest.Create(url + "pluginUpdate");
request.Method = "POST";
//create a byte array of the current version
byte[] requestContentTemp = version.ToByteArray();
int count = 0;
for (int i = 0; i < requestContentTemp.Length; i++) {
    if (requestContentTemp[i] != 0) {
        count++;
    }
}
byte[] requestContent = new byte[count];
for (int i = 0, j = 0; i < requestContentTemp.Length; i++) {
    if (requestContentTemp[i] != 0) {
        requestContent[j] = requestContentTemp[i];
        j++;
    }
}

//send the current version
request.ContentLength = requestContent.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(requestContent, 0, requestContent.Length);
dataStream.Close();

//get and read the response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] responseBytes = new byte[response.ContentLength];
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
responseStream.Close();
response.Close();

//if the response containd a single 0 we are up-to-date, otherwise write the content of the response to file
if (responseBytes[0] != 0 || response.ContentLength > 1) {
    BinaryWriter writer = new BinaryWriter(File.Open(Path.Combine(Directory.GetCurrentDirectory(), "ServerPlugins", "PointAwarder.dll"), FileMode.Create));
    writer.BaseStream.Write(responseBytes, 0, responseBytes.Length);
    writer.Close();
    TShockAPI.Commands.HandleCommand(TSPlayer.Server, "/reload");
}

The byte array responseBytes on the client should be identical to the array UpdateRequestResponse on the server, but it isn't. after about 4000 bytes every byte after that is set to 0 rather than what it should be (responseBytes[3985] is the last non-zero byte).

Does this happen because httpWebRequest has a size limit? I can't see any bug in my code that could be causing it and the same code works in other instances where I only have to pass around short sequences of data (less than 100 bytes).

The MSDN pages don't mention any size limit like this.

Answer 1

It's not that it has any artificial limit, this is a byproduct of the Streaming nature of what you're attempting to do. I have a feeling the following line is the offender:

responseStream.Read(responseBytes, 0, (int)response.ContentLength);

I've had this issue in the past (with TCP streams), it doesn't read all of the contents of the array, because they haven't all been sent over the wire yet. This is what I would try instead.

for (int i = 0; i < response.ContentLength; i++)
{
   responseBytes[i] = responseStream.ReadByte();
}

That way, it will make sure to read all the way until the end of the stream.

EDIT

usr's BinaryReader based solution is much more efficient. Here is the relevant solution:

BinaryReader binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
    byte[] buffer = new byte[bufferSize];
    int count;
    while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
        ms.Write(buffer, 0, count);
    responseBytes = ms.ToArray();
}

Answer 2

You are assuming that Read is reading as many bytes as you request. But the requested count is just an upper limit. You must tolerate reading small chunks.

You can use var bytes = new BinaryReader(myStream).ReadBytes(count); to read an exact number. Don't call ReadByte too often because that is very CPU intensive.

The best solution would be to step away from the fairly manual HttpWebRequest and use HttpClient or WebClient . All of this is automated for you and you get back a byte[] .