Why isn't my C# web request returning anything?

Question

I am hitting a URL that returns a long JSON set (REALLY long, 20 million characters). Just pasting the URL into Chrome, it takes about 3 minutes to return the full result set. Whatever the default settings in Chrome are, it prompts me several times to either Kill the page or Wait. But the page will return after several minutes.

I'm running this from SSIS with a script task. I'm not very familiar with C#. I copied/pasted this code from a sample:

{

    HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
    httpWReq.Method = "GET";
    httpWReq.ContentType = "application/json";
    httpWReq.Timeout = 300000;
    HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
    RootObject jsonResponse = null;

    try
    {
        //Get the stream of JSON
        Stream responseStream = httpWResp.GetResponseStream();

        //Deserialize the JSON stream
        using (StreamReader reader = new StreamReader(responseStream))
        {
            //Deserialize our JSON
            DataContractJsonSerializer sr = new DataContractJsonSerializer(typeof(RootObject));
            jsonResponse = (RootObject)sr.ReadObject(responseStream);
        }
    }
    //Output JSON parsing error
    catch (Exception e)
    {
        FailComponent(e.ToString());
    }
    return jsonResponse;

I am 110% positive that the wURL string is a valid JSON endpoint. When I step through my code, it waits maybe 15 seconds on this line:

HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();

... and then returns without error... but it doesn't populate what I'd expect into httpWResp (ContentLength = -1). When it gets to:

jsonResponse = (RootObject)sr.ReadObject(responseStream); 

... jsonResponse holds my pre-defined json container object, set to null. There are thousands of json arrays returned from my URL.

I don't see any interesting attributes in responseStream that would indicate that it actually contains anything?

What am I missing here?

I can't post the actual URL because it's a private company URL.

=================================

EDIT: I tried a URL with a much shorter string, and it returned. So it appears to be something about the length. I ran the return value through a validator and it succeeded... so possibly a special character, but I'm thinking likely the length.

Answer 1

From the comments we now know that the response object you get back from GetResponse() has a StatusCode of OK and a ContentType of application/json;charset=UTF-8 -- indicating that the server has returned the data 'chunked' which is why the ContentLength = -1.

You should be able to use the ReadToEnd() method on the StreamReader , something like this:

//Deserialize the JSON stream
using (StreamReader reader = new StreamReader(responseStream))
    {
        string r = reader.ReadToEnd();

        //Deserialize our JSON
        DataContractJsonSerializer sr = new DataContractJsonSerializer(typeof(RootObject));
        MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(r));
        jsonResponse = (RootObject)sr.ReadObject(ms);
}