HttpWebRequest Response Stream only returns 64k of data

Question

We're using the Web API to provide a RESTful service to our application, which at the moment is using WPF and so requires a service client.

The problem we're having is that when the response size exceeds 64k, the underlying stream stops at 64k and throws an exception if more data is requested, despite the fact that ContentLength is greater (in the test case ~125k)

Here is our method that calls the service:

private HttpWebResponse CallServiceMethod<TDto>(Uri serviceURL, string method, out WebException requestExceptionDetail, TDto dto = null) where TDto : DTOBase
{
  HttpWebRequest request;
  HttpWebResponse response;
  Encoding enc;
  string encodedDto;
  byte[] encodedDtoArray;
  Stream writeStream;

  request = WebRequest.CreateHttp(serviceURL);
  request.Method = method;

  if (dto != null)
  {
    enc = UTF8Encoding.Default;
    encodedDto = String.Concat("=", JsonConvert.SerializeObject(dto));
    encodedDtoArray = enc.GetBytes(encodedDto);

    request.ContentLength = encodedDtoArray.Length;
    request.ContentType = "application/x-www-form-urlencoded";

    writeStream = request.GetRequestStream();
    writeStream.Write(encodedDtoArray, 0, encodedDtoArray.Length);
    writeStream.Close();
  }

  requestExceptionDetail = null;

  try
  {
    response = (HttpWebResponse)request.GetResponse();
  }
  catch (WebException wex)
  {
    response = (HttpWebResponse)wex.Response;
    requestExceptionDetail = wex;
  }

  return response;
}

...and here's a method that decodes the response, with some glue logic in between that passes the response through.

private TObjectType DecodeResponse<TObjectType>(HttpWebResponse webResponse)
{
  Encoding enc;
  StreamReader responseReader;
  string responseText;
  TObjectType retVal;

  // Obtain a string from the underlying response string
  enc = UTF8Encoding.Default;
  responseReader = new StreamReader(webResponse.GetResponseStream(), enc);
  responseText = responseReader.ReadToEnd();

  // Use JSon to deserialise the string
  retVal = JsonConvert.DeserializeObject<TObjectType>(responseText);

  return retVal;
}

We have also tried this sort of thing, but to no avail:

private TObjectType DecodeResponse<TObjectType>(HttpWebResponse webResponse)
{
  Encoding enc;
  StreamReader responseReader;
  string responseText;
  TObjectType retVal;
  char[] buffer;
  int bufferSize, startPos, totalLength, readLength;
  StringBuilder sbx;

  // Obtain a string from the underlying response string
  enc = UTF8Encoding.Default;
  responseReader = new StreamReader(webResponse.GetResponseStream(), enc);

  totalLength = (int)webResponse.ContentLength;
  bufferSize = 65536;
  startPos = 0;
  buffer = new char[bufferSize];

  sbx = new StringBuilder(totalLength);
  //webResponse.GetResponseStream().

  while (startPos < totalLength)
  {
    readLength = Math.Min(bufferSize, totalLength - startPos);
    responseReader.Read(buffer, startPos, readLength);
    sbx.Append(buffer, 0, readLength);

    startPos += readLength;
  }

  //responseText = responseReader.ReadToEnd();
  responseText = sbx.ToString();

  // Use JSon to deserialise the string
  retVal = JsonConvert.DeserializeObject<TObjectType>(responseText);

  return retVal;
}

Any help would be much appreciated ;-)

Answer 1

Further investigation revealed that HttpWebRequest has a static property called DefaultMaximumErrorResponseLength which is set by default to 65536.

It would conspire that the .NET framework includes this fiendish property which works as long as your response is text/plain, but if it requires any closure - such as XML or JSON - it breaks.

For anybody who needs more information, please refer to:

• http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.defaultmaximumerrorresponselength.aspx