控制台应用程序中的最大并行任务，以及调用 API 时如何处理“将内容复制到流时出错”

Question

I'm working on an integration that handles several large datasets (around 3M records or so, growing daily) which are pulling from a vendor OData API in chunks. When I run it/debug locally, the integration will run to completion without errors. However, as soon as I put it out on our PROD server, it will sometimes fail with "Error while copying content to stream" on a given chunk of data. I'm having trouble figuring out the reason, and could use some help.

Each request is wrapped in a using statement and should be disposing of itself cleanly. Usually, it does. Again, this all works perfectly on my local machine, but the server has a bit of trouble after it's into pulling millions of records. I have added extra logging at virtually every step to catch the specific error, but it's still not clear how to handle (or avoid) this error. It isn't consistent at all... totally intermittent and inconsistent.

The code where the error throws:

using (WebRequestHandler webRequestHandler = new WebRequestHandler())
{
    using (HttpClient httpClient = new HttpClient(webRequestHandler))
    {
        httpClient.Timeout = TimeSpan.FromMinutes(15);

        ConfigureJsonClient(httpClient, syncConfig.ApiEndpoint, syncConfig.ApiAuthKey);

        apiCommand = "http://foo/with/filtering";

        responseMessage = Task.Run(async () => await httpClient.GetAsync(apiCommand)
            .ConfigureAwait(true)).Result;
    }
}

The error:

API Error Occurred - RETRYING... Foo: 2 | API CALL: https://foo... | EXCEPTION: One or more errors occurred. | INNER EXCEPTION: Error while copying content to a stream. | STACKTRACE:    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at FooNamespace.Processor.GetDataFromApi(SyncConfig syncConfig, String collection, List`1 nonDatedCollectionList, Int32 skipIndex, Int32 retryCount, Int32 taskId)

I am also trying to run as many parallel tasks as I can to pull these large datasets in pre-defined chunks (data pages) using skip/take. No matter what I do, even locally, the max number of Tasks I can spin up in parallel is 40. Is there a reason for this limitation?

Is it possible the parallel tasks are conflicting with each other somehow when hitting the API? These should appear as completely separate, isolated calls...

Any and all help is much appreciated.

Answer 1

I've got a similar exception Error while copying content to a stream . On the inner exception I've got System.IO.IOException: The decryption operation failed . I found that the server is sending the content compressed without the header content-encoding: gzip . Change de default value of the AutomaticDecompression property on HttpClientHandler solve the problem.

HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.AutomaticDecompression = DecompressionMethods.All;
using (var client = new HttpClient(httpClientHandler))