Multiple kestrel stateless service instances running on different ports in service fabric

Question

I have created a Stateless service in Visual Studio 2019 using the sample defaults (Create new project -> Service Fabric Application -> name the project and solution -> Stateless ASP.NET Core service).

Running the weather forecast application with no changes works fine, I can see the application running in the Service Fabric explorer and see my 5 healthy (local development) nodes and 1 green application.

If I modify Local.5Node.xml to change the InstanceCount from 1 to -1

<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/SampleApp" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="SampleApp_ASPNETCORE_ENVIRONMENT" Value="Development" />
    <Parameter Name="SampleApp_InstanceCount" Value="-1" />
  </Parameters>
</Application>

I can see under the single partition five instances get created, yet four of them are stuck in 'InBuild':

And in my running application in VS2019 the Diagnostics window shows processes continually failing to startup:

Upon further inspection, I see that the new instances are trying to be spun up on the same port.

How does port assignment work in Service Fabric with multiple instances of stateless web services? Does it require a gateway with a reverse proxy to discover each instance? Is autoscaling a stateless web service possible in Service Fabric without custom solutions?

Answer 1

Kestrel doesn't support port sharing, so as soon as one service claims the port, others running on the same machine (here: your dev machine) cannot.

The simplest workaround, is to create 1 instance of your service when running on your dev cluster. Another way to work around this, is to use (random) SF assigned ports locally:

Omit the Endpoint configuration in ServiceManifest.xml entirely, and don't pass an endpoint name to the KestrelCommunicationListener constructor

Next, use a (eg the built-in ) reverse proxy to communicate with your services.

More info here