Httpclient计算多个url远程文件大小

Question

我想计算 1000+ url 的总文件大小。 在任务运行时，我想显示当前数量和总数。 但是如果我添加您可以在我的代码中看到的信息，我会收到错误 ssl。

这是我得到的错误：

    "Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)"

您能否建议显示计数的最佳方式是什么？

 #region GET REMOTE FILE SIZE
public static async Task<string> GetRemoteFileSize(this List<string> urls) {
    long sum = 0;
    long localSum = 0;
    var tasks = new List<Task<long?>>();
    for(int i = 0; i < urls.Count; i++) {
        var url = urls[i];
        tasks.Add(GetTotalBytes(url));
        //if i will add here the counts, it's not getting the actual count
        Console.Title = $"{i} of {urls.Count}";
    }
    
    for (int i = 0; i < tasks.Count; i++) {
        Task<long?> task = tasks[i];
        localSum += Convert.ToInt32(await task.ConfigureAwait(false));
        Console.Title = $"{i} of {urls.Count}"; //if I will add this, I'm getting the SSL error even if I already added this " ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }," to my httpclient handler
    }
    await Task.WhenAll(tasks.ToArray()).ConfigureAwait(false);
    return Interlocked.Add(ref sum, localSum).FormatFileSize();
}
private static async Task<long?> GetTotalBytes(string url) {

    Uri uri = new Uri(url, UriKind.Absolute);
    uri.SetServiceManagerConnection();

    using var request = new HttpRequestMessage(HttpMethod.Head, uri);
    request.AddAllCommonHeaders(uri);
    using var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
    HttpStatusCode statusCode = response.StatusCode;

    if (!response.IsSuccessStatusCode) {
        throw new InvalidDataException($"Error occured: {statusCode} {(int)statusCode}");
    }

    return response.Content.Headers.ContentLength.GetValueOrDefault();
}
#endregion GET REMOTE FILE SIZE

// 我使用的 class

static string FormatBytes(this long bytes) {
        var unit = 1024;
        if (bytes < unit) { return $"{bytes} B"; }
        var exp = (int)(Math.Log(bytes) / Math.Log(unit));
        return $"{bytes / Math.Pow(unit, exp):F2} {("KMGTPE")[exp - 1]}B";
    }
    public static string FormatFileSize(this long bytes) => bytes.FormatBytes();

 public static void SetServiceManagerConnection(this Uri uri) {
        var sp = ServicePointManager.FindServicePoint(uri);
        sp.ConnectionLimit = 20; //default 2 //The number of connections per end point.
        sp.UseNagleAlgorithm = false; //Nagle’s algorithm is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network
        sp.Expect100Continue = false; //save bandwidth before sending huge object for post and put request to ensure remote end point is up and running.
        sp.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;//60 * 1000; // 1 minute
    }

更新：这个是有效的：

  static HttpClient Client { get; set; } = new(new HttpClientHandler { 
        Proxy = null, 
        UseProxy = false,
        SslProtocols = (SslProtocols)(SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13)
    });

Answer 1

更新：当 HTTPS 出现 TLS 握手错误时，此错误很典型。 您应该指定 SecurityProtocol

System.Net.ServicePointManager.SecurityProtocol = 
SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;

参考： https://docs.microsoft.com/en-us/dotnet/api/system.net.securityprotocoltype

Answer 2

正如其他人所提到的，您会收到 TLS 错误。

一旦你解决了这个问题，你正在做的操作的一个更快的方法将是 HTTP HEAD ，这只是返回给你的标题，你可以在其中看到大小。

请注意：并非所有服务器都允许HEAD ，特别是 AWS 例如。

此问题的答案显示如何使用HttpClient执行此操作： HTTP HEAD request with HttpClient in .NET 4.5 和 C#