ServiceFabric stateless WebApi automatically retries 503 and 404 responses

Question

We have a ServiceFabric stateless WebAPI frontend built on top of do.net5. I have implemented following exception handling filter for it:

    public class OrderServiceRetryFilter : IExceptionFilter
    {
        public void OnException(ExceptionContext context)
        {
            var exc = context.Exception;
            if (exc is AggregateException ae && (
                ae.InnerException is OrdersNotFetchedException onfe))
            {
                context.HttpContext.Response.Headers.Add("Retry-After", "2");
                var result = new ObjectResult(onfe.Message) { StatusCode = 591 };
                context.Result = result;
                context.ExceptionHandled = true;
            }
            if (exc is AggregateException ate && (
                ate.InnerException is System.TimeoutException toex))
            {
                context.HttpContext.Response.Headers.Add("Retry-After", "1");
                var result = new ObjectResult(toex.Message) { StatusCode = 504 };
                context.Result = result;
                context.ExceptionHandled = true;
            }
            if (exc is AggregateException anfe && (
                anfe.InnerException is OrderNotFoundException onf))
            {
                var result = new NotFoundObjectResult(onf.Message);
                context.Result = result;
                context.ExceptionHandled = true;
            }
        }
    }

If the stateful backend services throw an exception this filter will find the inner exception and return the correct status code (591, 504, 404) for the HTTP query.

Now, if the backend service throws a OrdersNotFetchedException the status code is set to 591 and the client will get it. I am using our own 591 because returning 503 would cause something to retry the call. This retry happens also in the case of 404. If I make a GET call, that will result in 404, from Postman, it will eventually just time out. Debugging the code shows that the code constantly returns to the OnException method that returns 404. If I change the error code to 592 during debug it will return that result code to the calling client without retries.

Something, somewhere, and I think it is the ServiceFabric, is retrying the simple API call if it returns 503 or 404. Where can I disable this kind of behavior or am I doing something against the way public facing Web APIs are designed with ServiceFabric?

This is how I start the Kestrel server:

        private IWebHost BuildWebHost(string url, AspNetCoreCommunicationListener listener)
        {
            ServiceEventSource.Current.ServiceMessage(Context, $"Starting Kestrel on {url}");

            var webHost = new WebHostBuilder()
                .UseKestrel()
                .ConfigureServices(
                    services => services
                        .AddSingleton(Context)                     
                        .AddSingleton(ServiceFabricRemoting.CreateServiceProxy<IOrderService>(new Uri($"{ServiceFabricRemoting.GetFabricApplicationName()}/MyApp.OrderService"), new MyLogger(Context), 1))                        
                        
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseStartup<Startup>()
                .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.UseUniqueServiceUrl)
                .UseUrls(url)
                .Build();

            HandleWebHostBuilt(webHost);

            return webHost;
        }

Answer 1

According to https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reverseproxy#special-handling-for-port-sharing-services the correct answer is to add X-ServiceFabric: ResourceNotFound header to the response to bypass the retry in the case of 404.

So, in this case:

context.HttpContext.Response.Headers.Add("X-ServiceFabric", "ResourceNotFound");