Google API BatchRequest：已建立的連接已被主機中的軟件中止

Question

我正在測試InsertAllRequest的Google BatchRequest（C＃）。 一旦一個批次達到60個以上的請求（〜30,000個bigquery表行/總共> 10,880,366 Http ContentLength），就會發生以下異常。 關閉防火牆時也是如此。 我在網上找到的解決方案（例如關閉http keep-alive）在這種情況下不起作用，因為我無法控制API如何使用HttpClient。

    System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
   at System.Net.GZipWrapperStream.EndRead(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.EndRead(IAsyncResult asyncResult)
   at System.Net.Http.StreamToStreamCopy.StartRead()
   --- End of inner exception stack trace ---
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Google.Apis.Requests.BatchRequest.<ExecuteAsync>d__3.MoveNext() in c:\ApiaryDotnet\default\Src\GoogleApis\Apis\Requests\BatchRequest.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at BigData.BigQuery.API.BigQueryHelper.<InsertBatchAsync>d__1f.MoveNext() in c:\Users\fionazhao\Documents\BigDataCode\Framework\BigData\BigQuery\API\BigQueryHelper.cs:line 197
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

這是引發此異常的Google API代碼：

 public async Task ExecuteAsync(CancellationToken cancellationToken)
    {
        if (Count < 1)
            return;

        ConfigurableHttpClient httpClient = service.HttpClient;

        var requests = from r in allRequests
                       select r.ClientRequest;
        HttpContent outerContent = await CreateOuterRequestContent(requests).ConfigureAwait(false);
        var result = await httpClient.PostAsync(new Uri(batchUrl), outerContent, cancellationToken)
            .ConfigureAwait(false);

        result.EnsureSuccessStatusCode();

        // Get the boundary separator.
        const string boundaryKey = "boundary=";
        var fullContent = await result.Content.ReadAsStringAsync().ConfigureAwait(false);
        var contentType = result.Content.Headers.GetValues("Content-Type").First();
        var boundary = contentType.Substring(contentType.IndexOf(boundaryKey) + boundaryKey.Length);

        int requestIndex = 0;
        // While there is still content to read, parse the current HTTP response.
        while (true)
        {
            cancellationToken.ThrowIfCancellationRequested();

            var startIndex = fullContent.IndexOf("--" + boundary);
            if (startIndex == -1)
            {
                break;
            }
            fullContent = fullContent.Substring(startIndex + boundary.Length + 2);
            var endIndex = fullContent.IndexOf("--" + boundary);
            if (endIndex == -1)
            {
                break;
            }

            HttpResponseMessage responseMessage = ParseAsHttpResponse(fullContent.Substring(0, endIndex));

            if (responseMessage.IsSuccessStatusCode)
            {
                // Parse the current content object.
                var responseContent = await responseMessage.Content.ReadAsStringAsync().ConfigureAwait(false);
                var content = service.Serializer.Deserialize(responseContent,
                    allRequests[requestIndex].ResponseType);

                allRequests[requestIndex].OnResponse(content, null, requestIndex, responseMessage);
            }
            else
            {
                // Parse the error from the current response.
                var error = await service.DeserializeError(responseMessage).ConfigureAwait(false);
                allRequests[requestIndex].OnResponse(null, error, requestIndex, responseMessage);
            }

            requestIndex++;
            fullContent = fullContent.Substring(endIndex);
        }
    }

Answer 1

剛好超過10MB，這是流插入的請求大小限制。 為這些限制調整代碼：

最大行大小：1 MB
HTTP請求大小限制：10 MB
每秒最大行數：每個表每秒100,000行。 超過此金額將導致quota_exceeded錯誤。
每個請求的最大行數：500
每秒最大字節數：每張表每秒100 MB。 超過此金額將導致quota_exceeded錯誤。

https://cloud.google.com/bigquery/streaming-data-into-bigquery