ASP.NET Core 3.1 app running on Azure App Service throwing EPIPE errors for 1.6 MB json payload

Question

I have a simple ASP.NET Core 3.1 app deployed on an Azure App Service, configured with a .NET Core 3.1 runtime. One of my endpoints are expected to receive a simple JSON payload with a single "data" property, which is a base64 encoded string of a file. It can be quite long, I'm running into the following issue when a the JSON payload is 1.6 MBs.

On my local workstation, when I call my API from Postman, everything's working as expected, my breakpoint in the Controller's action method is reached, the data is populated, all good - it's only when I deploy (via Azure DevOps CICD Pipelines) the app to the Azure App Service. Whenever trying to call the deployed API from Postman, no HTTP response is received, but this: "Error: write EPIPE".

I've tried modifying the web.config to include both a maxRequestLength and maxAllowedContentLength properties:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
      <system.web>
          <httpRuntime maxRequestLength="204800" ></httpRuntime>
          </system.web>
    <system.webServer>
            <security>
        <requestFiltering>          
            <requestLimits maxAllowedContentLength="419430400" />
        </requestFiltering>
    </security>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\MyApp.API.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" />
    </system.webServer>
  </location>
</configuration>

In the app's code, I've added to the Startup.cs:

services.Configure<IISServerOptions>(options => {
    options.MaxRequestBodySize = int.MaxValue;
});

In the Program.cs, I've added:

.UseKestrel(options => { options.Limits.MaxRequestBodySize = int.MaxValue; })

In the controller, I've tried both of these attributes: [DisableRequestSizeLimit], [RequestSizeLimit(40000000)]

However, nothing's working so far - I'm pretty sure it has to be something configured on the App Service itself, not in my code, as locally everything's working. Yet, nothing so far helped in the web.config

Answer 1

It was related to the fact that in my App Service, I had to allow incoming client certificates, in the Configuration - turns out client certificates and large payloads don't mix well in IIS (apparently for more than a decade now): https://docs.microsoft.com/en-us/archive/blogs/waws/posting-a-large-file-can-fail-if-you-enable-client-certificates

None of the proposed workarounds in the above blog post fixed my issue, so I had to workaround: I've created an Azure Function (still using .NET Core 3.1 as a runtime stack) with a Consumption Plan, which is able to receive both the large payload and the incoming client certificate (I guess it doesn't use IIS under the hood?).

In my original backend, I added the original API's route to the App Service's "Certificate exclusion paths", to not get stuck waiting and timing out eventually with "Error: write EPIPE".

I've used Managed Identity to authenticate between my App Service and the new Azure Function (through a System Assigned identity in the Function).

The Azure Function takes the received certificate, and adds it to a new "certificate" property in the JSON body, next to the original "data" property, so my custom SSL validation can stay on the App Service, but the certificate is not being taken from the X-ARR-ClientCert header, but from the received payload's "certificate" property.

The Function:

#r "Newtonsoft.Json"
using System.Net;
using System.IO;
using System.Net.Http;
using System.Text;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
using System.Security.Cryptography.X509Certificates;

private static HttpClient httpClient = new HttpClient();

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
    var requestBody = string.Empty;
    using (var streamReader =  new StreamReader(req.Body))
    {
        requestBody = await streamReader.ReadToEndAsync();
    }

    dynamic deserializedPayload = JsonConvert.DeserializeObject(requestBody);
    var data = deserializedPayload?.data;
    
    var originalUrl = $"https://original-backend.azurewebsites.net/api/inbound";
    var certificateString = string.Empty;

    StringValues cert;
    if (req.Headers.TryGetValue("X-ARR-ClientCert", out cert))
    {
        certificateString = cert;
    }

    var newPayload = new {
        data = data,
        certificate = certificateString
    };

    var response = await httpClient.PostAsync(
        originalUrl,
        new StringContent(JsonConvert.SerializeObject(newPayload), Encoding.UTF8, "application/json"));

    var responseContent = await response.Content.ReadAsStringAsync();

    try
    {
        response.EnsureSuccessStatusCode();
        return new OkObjectResult(new { message = "Forwarded request to the original backend" });
    }
    catch (Exception e)
    {
        return new ObjectResult(new { response = responseContent, exception = JsonConvert.SerializeObject(e)})
        {
            StatusCode = 500
        };
    }
}