'Stream was already consumed' error using Polly to retry requests in ASP.NET Core

Question

I have an ASP.NET Core 3.1 Web API service that is receiving a web request, doing some manipulations to it, and then passing it on to a backend service and returning the response synchronously. It was working fine but I wanted to introduce some retry logic to those backend requests in case some blips occurred.

I'm using a typed HttpClient and attempting to use Polly to implement the retry logic: https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#using-polly-with-ihttpclientfactory

When the backend service works, everything seems to be fine but unfortunately, whenever my backend returns an error like a 500 Internal Server Error, I get the following exception:

Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
   at System.Net.Http.StreamContent.PrepareContent()
   at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at MyProject.MyController.Routing.HttpMessageRouter.SendRequestAndTrackTiming(Func`1 action, String destinationID) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 59
   at MyProject.MyController.Routing.HttpMessageRouter.SendNewRequest(IMessageWrapper`1 message) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 33
   at MyProject.MyController.Controllers.MyControllerController.Resource(String destinationId) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Controllers/MyControllerController.cs:line 151
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Prometheus.HttpMetrics.HttpRequestDurationMiddleware.Invoke(HttpContext context)
   at Prometheus.HttpMetrics.HttpRequestCountMiddleware.Invoke(HttpContext context)
   at Prometheus.HttpMetrics.HttpInProgressMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

The code I have to send the request is as follows. This first method actually just utilizes a delegate so I can transparently add some metrics gathering around the typed HttpClient call. The typed HttpClient is called RoutingClient in this code:

public async Task<IActionResult> SendNewRequest(IMessageWrapper<HttpRequestMessage> message)
{
    HttpResponseMessage destinationResponse = await SendRequestAndTrackTiming(() => _client.SendAsync(message.Message), message.DestinationID);
    return CreateSerializeableResponseMessage(destinationResponse);
}

private ResponseMessageResult CreateSerializeableResponseMessage(HttpResponseMessage httpResponse)
{
    ResponseMessageResult responseMessage = new ResponseMessageResult(httpResponse);
    IOutputFormatter[] formattersList = { new HttpResponseMessageOutputFormatter() };
    FormatterCollection<IOutputFormatter> formattersCollection = new FormatterCollection<IOutputFormatter>(formattersList);
    responseMessage.Formatters = formattersCollection;
    return responseMessage;
}

private async Task<HttpResponseMessage> SendRequestAndTrackTiming(Func<Task<HttpResponseMessage>> action, string destinationID)
{
    HttpResponseMessage response = null;

    Stopwatch stopwatch = new Stopwatch();
    try
    {
        stopwatch.Start();
        response = await action();
        return response;
    }
    finally
    {
        stopwatch.Stop();
        HttpStatusCode statusCode = (response != null) ? response.StatusCode : HttpStatusCode.InternalServerError;
        _routedMessageMetricTracker.Histogram.Observe(destinationID, statusCode, stopwatch.Elapsed.TotalSeconds);
    }
}

This is my code in Startup.ConfigureServices() (actually this is in an extension method I defined):

public static IServiceCollection AddRoutingClient(this IServiceCollection services, RoutingSettings routingSettings)
{
    List<TimeSpan> retryTimeSpans = new List<TimeSpan>();

    // routingSettings.RetrySeconds is just an array of double values.
    foreach (double retrySeconds in routingSettings.RetrySeconds)
    {
        if (retrySeconds >= 0) retryTimeSpans.Add(TimeSpan.FromSeconds(retrySeconds));
    }

    services.AddHttpClient<IRoutingClient, RoutingClient>()
        .AddPolicyHandler((services, request) => HttpPolicyExtensions.HandleTransientHttpError()
            .WaitAndRetryAsync(retryTimeSpans, onRetry: (outcome, timespan, retryAttempt, context) =>
                {
                    services.GetService<ILogger<RoutingClient>>()?
                        .LogWarning($"Delaying for {timespan.TotalMilliseconds}ms, then making retry {retryAttempt}.");
                }
            ));

    return services;
}

I'd really like to use the Polly approach because it seems clean but I am not sure what I am doing wrong here. I must be doing something wrong because I would expect this to be a very common use-case for Polly that should be handled.

Answer 1

It turns out my problem was not caused by the code dealing with the response above. Instead, it was actually caused by my code that was manipulating the request .

I was reading the incoming request as an HttpRequestMessage object via the HttpContext.GetHttpRequestMessage() method and I attempted to re-use that same object to pass on to the backend service via a call to my typed HttpClient . However, the content stream for that request is read-once so I had to make a copy of that HttpRequestMessage as described in this answer to another post: https://stackoverflow.com/a/34049029/1221718

Here is a slightly more verbose version of that answer's code:

private static async Task<HttpRequestMessage> CloneHttpRequestMessageAsync(HttpRequestMessage request)
{
    HttpRequestMessage copyOfRequest = new HttpRequestMessage(request.Method, request.RequestUri);

    // Copy the request's content (via a MemoryStream) into the cloned object
    var ms = new MemoryStream();
    if (request.Content != null)
    {
        await request.Content.CopyToAsync(ms).ConfigureAwait(false);
        ms.Position = 0;
        copyOfRequest.Content = new StreamContent(ms);

        // Copy the content headers
        if (request.Content.Headers != null)
        {
            foreach (var h in request.Content.Headers)
            {
                copyOfRequest.Content.Headers.Add(h.Key, h.Value);
            }
        }
    }

    copyOfRequest.Version = request.Version;

    foreach (KeyValuePair<string, object> prop in request.Properties)
    {
        copyOfRequest.Properties.Add(prop);
    }

    foreach (KeyValuePair<string, IEnumerable<string>> header in request.Headers)
    {
        copyOfRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);
    }

    return copyOfRequest;
}