Servce Fabric Reverse Proxy Integration in ASP.NET Core 404 Response

Question

I'm working on an implementation of ICommunicationClient and accompanying stuff for HTTP protocol communication which should be compatible with SF reverse proxy. The most subtle part for me is retry policy. According to Azure docs for 404 errors reverse proxy relies on X-Service-Fabric header to be returned from web service when deciding if it should retry.

ASP.NET Core provides middleware for integration with a reverse proxy which adds X-Service-Fabric header to every 404 response.

Assume we have the scenario when ServicePartitionClient cached the endpoint for a stateless service listening on port 3001. At some point, this service may be moved to another node. On the first node, the Service Fabric runtime allocates a different service with its own endpoints but using the same middleware and listening on the same 3001 port.

When a client tries to call the original service at its old (cached) address it will receive 404 response containing the X-Service-Fabric header. According to reverse proxy policies it shouldn't retry, but for me, it seems like the client will stay connected to the wrong service forever and won't attempt to re-resolve the endpoint.

I can't find any information about this case in the documentation, did I miss something here? Is it safe to rely on this standard middleware and don't do retry attempts on 404 errors with X-Service-Fabric: ResourceNotFound header?

Answer 1

In the described case the communication client will be invalidated by staying connected to the wrong service. It is recommended by Microsoft to use unique URL prefixes for services with dynamically-assigned port to handle those scenarios.

In ASP.NET Core programmers can leverage ServiceFabricMiddleware which checks URL prefixes and returns 410 Gone if they don't match. Then HTTP implementation of ICommunicationClient can retry with re-resolving endpoint only for 410 responses and don't perform any retries on 404 response with X-Service-Fabric: ResourceNotFound header if reverse proxy integration is enabled.

Answer 2

In your given scenario, when your Client encounters a 404, the X-Service-Fabric:ResourceNotFound header isn't the only attribute your code could examine when deciding whether or not to retry some operation.

To simplistically address your concern that your client won't be able to tell the difference between a "friendly" node and a "newly arrived" node, and since you're already working with http headers, you could add a custom HTTP header to outgoing responses to identify that a request comes from your application. When the client receives a 404, you can simply check for the presence of your custom header to answer the question of whether or not it's a "legit" retry. Of course, adding a custom HTTP header just for the sake of this validation check may be more of a global solution to a local problem. Ed: goes without saying that this should not be used to make security decisions by the application

A more elegant and complex means of accomplishing the same would be to deduce the same result using a different combination of HTTP headers and response attributes (eg, see if some other headers are expected/unexpected), but this could also be a hyper-local solution to the problem.