C# MVC - 无法使用 webClient 从服务器 WebAPI url 检索 JSON 数据

Question

I'm trying to load JSON data from WEB API. URL is working fine if I request data from any browser. The same URL is not working if I'm trying to fetch the data using webClient. It throws below error:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) +111    
[IOException: Unable to read data from the transport connection: A connection attempt failed     because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]

System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) +299
    [WebException: The underlying connection was closed: An unexpected error occurred on a receive.]
       System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request) +298
       System.Net.WebClient.DownloadString(Uri address) +106

I found so may stack overflow similar questions and implemented TSL security protocol and header changes but still its not working.

Below is my code:

 PCR IWebClientServices<PCR>.PrepareWebClient(string PreparedURL)
    {
        try
        {
            //this code bypass the SSL exception
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | 
                                                   SecurityProtocolType.Tls11 | 
                                                   SecurityProtocolType.Tls;
            //now we will invoke the webClient
            var webClient = new WebClient();
            webClient.Headers.Add(HttpRequestHeader.Cookie, "cookievalue");
            webClient.Headers.Add(HttpRequestHeader.UserAgent, "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0");
            var json = webClient.DownloadString(PreparedURL);
            return JsonConvert.DeserializeObject<PCR>(json);
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
            return null;
        }
    }

I'm stuck here as this code works sometimes fine, sometimes not works at all. I'm clueless whats going wrong, but I think there is something I'm missing while requesting the data like browser request but server is catching as some fishy request and not responding. But if I browse URL from browser its sending the data.

Answer 1

For supporting TLS 1.2 connections, .NET 4.7.2 should be targetted.

When you want to make sure the application is running at a certain runtime, make sure the following section is present in your config file:

App.config

<configuration>

 <startup>
   <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
 </startup>
</configuration>

Web.config

 <system.web>
   <compilation debug="true" targetFramework="4.7.2" /> 
   <httpRuntime targetFramework="4.7.2" maxRequestLength="102400" />
 </system.web>
</configuration>

Additional remark:

I'm not sure if this could be a problem, but webclient is a deprecated way of calling webservices, you should prefer httpclient to make your requests.

Answer 2

Finally I was able to fix the issue. Issue was because of server has upgraded its SSL to TLS 1.2 with AES128 certificate. I changed webclient to httpclient & I avoided using block.

using(var client = new HttpClient())
{
    //I removed this code as using block code serves only one time
    // And was failing for next subsequent request
}

I searched a lot and found that if you are using HttpClient in using block then the object will be disposed but TCP connection will remain open. It will not get closed. So above approach was working for single request and next subsequent requests were failing because of this issue.

So I followed below approach and finally my code is working like charm now.

I initialized the HttpClient object as static in my class

private static HttpClient httpClient = new HttpClient();

Then I called this for my generic classes, passed all the info to it and its returning me the desired results.

Below is the full working code:

public class MyWebClientServices
{
    private static HttpClient httpClient = new HttpClient();

    public T PrepareWebClient<T>(string preparedURL)
    {
        try
        {
          // As my server is using TLS1.2 I kept only that protocol
          ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

          httpClient.DefaultRequestHeaders.Accept.Clear();
          httpClient.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualtiyHeaderValue("application/json");
          var json = httpClient.GetStringAsync(preparedURL).Result;
          //here T is generic PCR class. I have created generic classes for which I need to invoke httpClient & their respective urls as I want to use this same code for all.
          //If your class is not generic then instead of T you can use your class
          T tJson = JsonConvert.DeserializeObject<T>(json);
          return (T)Convert.ChangeType(tJson, typeof(T));
         //finally my issue has been fixed and I'm getting data on all subsequent requests too
    }
    catch(Exception e)
    {
        throw e;
    }

}//end of MyWebClientServices class

I had referred below blog which has all the information related to this issue: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/

2) If you are creating singleton pattern for httpClient and sharing across then it has also serious behavior.

That is HttpClientHandler creates a connection group (named with its hashcode) and does not close the connections in the group until getting disposed. This basically means the DNS check never happens as long as a connection is open.

A naive solution would be to dispose the HttpClient (hence the HttpClientHandler) every time you use it.

Another solution is to set ConnectionClose property of DefaultRequestHeaders on your HttpClient:

var client = new HttpClient();
client.DefaultRequestHeaders.ConnectionClose = true;

This will set the HTTP's keep-alive header to false so the socket will be closed after a single request. It turns out this can add roughly extra 35ms (with long tails, ie amplifying outliers) to each of your HTTP calls preventing you to take advantage of benefits of re-using a socket. So what is the solution then?

Now to fix it, all we need to do is to get hold of the ServicePoint object for the endpoint by passing the URL to it and set the ConnectionLeaseTimeout:

var sp = ServicePointManager.FindServicePoint(new Uri("http://foo.bar/baz/123?a=ab"));
sp.ConnectionLeaseTimeout = 60*1000; // 1 minute

The second part I have not used but in case if any one want to use singleton pattern then posting here. The more detail information can be obtained at below blog, important part I have posted here in case in future if link gets broken: http://byterot.blogspot.com/2016/07/singleton-httpclient-dns.html

I hope this will help someone who is stuck like me.