What is the best way to read GetResponseStream()?

Question

What is the best way to read an HTTP response from GetResponseStream ?

Currently I'm using the following approach.

Using SReader As StreamReader = New StreamReader(HttpRes.GetResponseStream)
   SourceCode = SReader.ReadToEnd()
End Using

I'm not quite sure if this is the most efficient way to read an http response.

I need the output as string, I've seen an article with a different approach but I'm not quite if it's a good one. And in my tests that code had some encoding issues with in different websites.

How do you read web responses?

Answer 1

I use something like this to download a file from a URL:

if (!Directory.Exists(localFolder))
{
    Directory.CreateDirectory(localFolder);   
}


try
{
    HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(Path.Combine(uri, filename));
    httpRequest.Method = "GET";

    // if the URI doesn't exist, an exception will be thrown here...
    using (HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse())
    {
        using (Stream responseStream = httpResponse.GetResponseStream())
        {
            using (FileStream localFileStream = 
                new FileStream(Path.Combine(localFolder, filename), FileMode.Create))
            {
                var buffer = new byte[4096];
                long totalBytesRead = 0;
                int bytesRead;

                while ((bytesRead = responseStream.Read(buffer, 0, buffer.Length)) > 0)
                {
                    totalBytesRead += bytesRead;
                    localFileStream.Write(buffer, 0, bytesRead);
                }
            }
        }
    }
}
catch (Exception ex)
{
    // You might want to handle some specific errors : Just pass on up for now...
    // Remove this catch if you don't want to handle errors here.
    throw;
}

Answer 2

My simple way of doing it to a string. Note the true second parameter on the StreamReader constructor. This tells it to detect the encoding from the byte order marks and may help with the encoding issue you are getting as well.

string target = string.Empty;
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=583");

HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
try
{
  StreamReader streamReader = new StreamReader(response.GetResponseStream(),true);                
  try
  {
    target = streamReader.ReadToEnd();
  }
  finally
  {
    streamReader.Close();
  }
}
finally
{
  response.Close();
}

Answer 3

Maybe you could look into the WebClient class. Here is an example :

using System.Net;

namespace WebClientExample
{
    class Program
    {
        static void Main(string[] args)
        {
            var remoteUri = "http://www.contoso.com/library/homepage/images/";
            var fileName = "ms-banner.gif";
            WebClient myWebClient = new WebClient();
            myWebClient.DownloadFile(remoteUri + fileName, fileName);
        }
    }
}

Answer 4

In powershell, I have this function:

function GetWebPage
{param ($Url, $Outfile)
    $request = [System.Net.HttpWebRequest]::Create($SearchBoxBuilderURL)
    $request.AuthenticationLevel = "None"
    $request.TimeOut = 600000     #10 mins 
    $response = $request.GetResponse() #Appending "|Out-Host" anulls the variable
    Write-Host "Response Status Code: "$response.StatusCode
    Write-Host "Response Status Description: "$response.StatusDescription
    $requestStream = $response.GetResponseStream()
    $readStream = new-object System.IO.StreamReader $requestStream
    new-variable db | Out-Host
    $db = $readStream.ReadToEnd()
    $readStream.Close()
    $response.Close()
    #Create a new file and write the web output to a file
    $sw = new-object system.IO.StreamWriter($Outfile)
    $sw.writeline($db) | Out-Host
    $sw.close() | Out-Host
}

And I call it like this:

$SearchBoxBuilderURL = $SiteUrl + "nin_searchbox/DailySearchBoxBuilder.asp"
$SearchBoxBuilderOutput="D:\ecom\tmp\ss2.txt"
GetWebPage $SearchBoxBuilderURL $SearchBoxBuilderOutput

Answer 5

You forgot to define "buffer" and "totalBytesRead":

using ( FileStream localFileStream = ....  
{  
    byte[] buffer = new byte[ 255 ];  
    int bytesRead;  
    double totalBytesRead = 0;  

    while ((bytesRead = .... 

Answer 6

I faced a similar situation:

I was trying to read raw response in case of an HTTP error consuming a SOAP service, using BasicHTTPBinding.

However, when reading the response using GetResponseStream() , got the error:

Stream not readable

So, this code worked for me:

try
{
    response = basicHTTPBindingClient.CallOperation(request);
}
catch (ProtocolException exception)
{
    var webException = exception.InnerException as WebException;

    var alreadyClosedStream = webException.Response.GetResponseStream() as MemoryStream;
    using (var brandNewStream = new MemoryStream(alreadyClosedStream.ToArray()))
    using (var reader = new StreamReader(brandNewStream))
        rawResponse = reader.ReadToEnd();
}