API调用因超出大量数据而超时

Question

I am having trouble in fetching XML data from through API in large amount. Has anyone a better solution in fetching XML data in ASP.net application?

Maybe the issue is might be garbage collection memory not sure but earlier it was not working for any user so I added manual memory release code and closing web response connection after each call.

Any suggestion to improve the fetching of large amount of data are welcome?

   public static XmlDocument LoadXmlFromUrl(string url, bool keepAlive = true, bool throwException = false, int timeout = 15000)
    {
        Log(url, "*** start ***" + url);

        ServicePointManager.DefaultConnectionLimit = 2000;
        string xmlAsText = null;
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
        req.KeepAlive = keepAlive;
        req.Timeout = timeout; // 30,000ms = 30 sec default
        try
        {
            bool retried = false;
        retry:
            try
            {
                using (HttpWebResponse webResponse = (HttpWebResponse)req.GetResponse())
                {
                    using (StreamReader responseStream = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8))
                    {
                        xmlAsText = responseStream.ReadToEnd();
                        responseStream.Close();
                    }
                    webResponse.Close();

                    Log(url, "stream reader (responseStream.ReadToEnd) completed to string.");
                }
            }
            catch (WebException webEx)
            {
                if (!retried)
                {
                    if (webEx.Status == WebExceptionStatus.Timeout || webEx.Status == WebExceptionStatus.KeepAliveFailure)
                    {
                        Log(url, "About to retry for url (6 sec wait): " + url);
                        System.Threading.Thread.Sleep(6000);
                        retried = true;
                        goto retry;
                    }
                }
                else
                {
                    throw webEx;
                }
            }
        }
        catch (Exception ex)
        {
            Log(url, "exception while loading responseStream.ReadToEnd. " + ex.ToString());
        }
        XmlDocument xdoc = null;
        if (!string.IsNullOrEmpty(xmlAsText))
        {
            try
            {
                xdoc = new XmlDocument();
                xdoc.LoadXml(xmlAsText);
                Log(url, "string added to xml (xdoc.LoadXml) completed.");
            }
            catch (Exception ex)
            {
                Log(url, "exception while loading xdoc.LoadXml. " + ex.ToString());
            }
        }
        return xdoc;
    }

Exception 1:

5/01/2017 12:27:01 p.m. exception while loading responseStream.ReadToEnd.     System.Net.WebException: The operation has timed out**

Exception 2:

6/01/2017 8:35:59 a.m. exception while loading responseStream.ReadToEnd. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Answer 1

Exception #1 may occur if

• request is huge. In this case you can try to increase value of HttpWebRequest.ReadWriteTimeout to something relatively big, for example, 3600000 (1 hour).

• value of header ContentLength does not match real response size and server does not force connection to close.

To track down the problem I would suggest you to set up tracing. You can follow https://msdn.microsoft.com/en-us/library/ty48b824(v=vs.110).aspx or find similar here on StackOverflow.

Exception #2 definitely occurs just because the response does not fit to memory allocated for application or its size exceeded 2Gb. (It must be really big response). The possible solution is to buffer stream to a file or process it on-fly (if possible). For example:

...
using (HttpWebResponse webResponse = (HttpWebResponse)req.GetResponse())
{
    String tempFile = Path.GetTempFileName();
    using (var stream = webResponse.GetResponseStream())
    {
        if (stream != null)
        {
            using (FileStream file = File.Create(tempFile))
            {
                stream.CopyTo(file);
            }
        }
    }

    Log(url, "stream reader (responseStream.ReadToEnd) completed to string.");
}