HttpWebRequest chunked/async POST

Question

Hi I want to upload some dynamic generated content to my web api. On the client I use the HttpWebRequest. The data should be uploaded sync and I want to write to the stream AFTER(!) I executed the HTTP-request.

(From server to client it does work fine, but from client to server i get some exceptions).

The client implementation looks like:

HttpWebRequest httpWebRequest = HttpWebRequest.Create(myUrl) as HttpWebRequest;
httpWebRequest.Method = "POST";
httpWebRequest.Headers["Authorization"] = "Basic " + ... ;
httpWebRequest.PreAuthenticate = true;

httpWebRequest.SendChunked = true;
//httpWebRequest.AllowWriteStreamBuffering = false; //does not help...
httpWebRequest.ContentType = "application/octet-stream";
Stream st = httpWebRequest.GetRequestStream();

Task<WebResponse> response = httpWebRequest.GetResponseAsync();

// NOW: Write after GetResponse()

var b = Encoding.UTF8.GetBytes("Test1");
st.Write(b, 0, b.Length);

b = Encoding.UTF8.GetBytes("Test2");
st.Write(b, 0, b.Length);

b = Encoding.UTF8.GetBytes("Test3");
st.Write(b, 0, b.Length);

st.Close();

var x = response.Result;
Stream resultStream = x.GetResponseStream();
//do some output...

I get exceptions (NotSupportedException: The stream does not support concurrent IO read or write operations.) on stream.write().

Why do I get here an exceptions. Some times the first writes worke and the late writes throw the exception. On the beginning the stream.CanWrite property is true, but after the first or second or thrird write it goes false... And then the exception is thrown on the next write.

Edit: Changing AllowWriteStreamBuffering did not help

Appendix: I found my problem. This problem is caused by the order of my code. I have to call it in this order:

• GetRequestStream (writing async to the stream) (the request is send to the server after the first write) then:
• GetResponseAsync()
• GetResponseStream()

I thought "GetResponseAsync" triggers the client to send the Request (for now the headers only). But it is not necessary because the request is already send after I write the first bits to the stream.

Second cause of my problems: Fiddler. (Fiddler currently only supports streaming of responses and not requests)

Answer 1

I found my problem.

The order of my code caused the problem.

The solution is to call it in this order:

• GetRequestStream (writing async to the stream) (the request is send to the server after the first write) then:
• GetResponseAsync()
• GetResponseStream()

My understanding is that "GetResponseAsync" triggers the client to send the request (for now the headers only), but I have discovered it to be an uneccesary step, because the request had already been sent after the first few bits had been written to the stream.

The second cause of my problems is Fiddler, but Fiddler only supports streaming of responses, and not requests.

The code acheived referencing, 'HttpWebRequest' class:

HttpWebRequest httpWebRequest = HttpWebRequest.Create("http://xxx") as HttpWebRequest;
httpWebRequest.Method = "POST";
httpWebRequest.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes("user:pw"));
httpWebRequest.PreAuthenticate = true;    
httpWebRequest.SendChunked = true;
httpWebRequest.AllowWriteStreamBuffering = false;
httpWebRequest.AllowReadStreamBuffering = false;    
httpWebRequest.ContentType = "application/octet-stream";

Stream st = httpWebRequest.GetRequestStream();

Console.WriteLine("Go");

try
{
    st.Write(buffer, 0, buffer.Length); //with the first write, the request will be send.
    st.Write(buffer, 0, buffer.Length);
    st.Write(buffer, 0, buffer.Length);

    for (int i = 1; i <= 10; i++)
    {        
        st.Write(buffer, 0, buffer.Length); //still writing while I can read on the stream at my ASP.NET web api

    }

}
catch (WebException ex)
{
    var y = ex.Response;

}
finally
{
    st.Close();

}

// Now we can read the response from the server in chunks

Task<WebResponse> response = httpWebRequest.GetResponseAsync();

Stream resultStream = response.Result.GetResponseStream();

byte[] data = new byte[1028];

int bytesRead;

while ((bytesRead = resultStream.Read(data, 0, data.Length)) > 0)
{
    string output = System.Text.Encoding.UTF8.GetString(data, 0, bytesRead);

    Console.WriteLine(output);

}

The code acheived referencing, 'HttpClient' class:

HttpClientHandler ch = new HttpClientHandler();

HttpClient c = new HttpClient(ch);    
c.DefaultRequestHeaders.TransferEncodingChunked = true;    
c.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(Encoding.ASCII.GetBytes("user:pw")));

Stream stream = new MemoryStream();

AsyncStream asyncStream = new AsyncStream(); // Custom implementation of the PushStreamContent with the method, "WriteToStream()".

PushStreamContent streamContent = new PushStreamContent(asyncStream.WriteToStream);

HttpRequestMessage requestMessage = new HttpRequestMessage(new HttpMethod("POST"), "http://XXX") { Content = streamContent };

requestMessage.Headers.TransferEncodingChunked = true;

HttpResponseMessage response = await c.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead);

// The request has been sent, since the first write in the "WriteToStream()" method.

response.EnsureSuccessStatusCode();

Task<Stream> result = response.Content.ReadAsStreamAsync();

byte[] data = new byte[1028];

int bytesRead;

while ((bytesRead = await result.Result.ReadAsync(data, 0, data.Length)) > 0)
{
    string output = System.Text.Encoding.UTF8.GetString(data, 0, bytesRead);

    Console.WriteLine(output);

}

Console.ReadKey();

Answer 2

You are using objects of an HttpWebRequest on multiple threads concurrently. response is a task that runs concurrently with your writes. That is clearly thread-unsafe.

Also, I don't see what you want to achieve. HTTP has a request-then-response model. The server can't (usually) send a single response byte before receiving the entire request. Theoretically, this is possible. But that is very unusual and likely not supported by the .NET BCL. This would only be supported by a (very unusual) custom HTTP server of yours.