Azure Service Bus Listener opening too many TCP connections (exhaustion)

Question

We have several Service Bus Listeners running as continuous Azure Webjobs inside App Services. All in all there are 12 listener-webjobs running on the same S1 App Service Plan. The environment is small, there are about ~1000-10000 messages per day in total sum. Recently we deployed a new Listener (a listener that periodically resends DLQ messages to the originating topic for up to 24h & 10 retries (exponential backoff)) and yesterday we got a TCP/IP exhaustion error message on the hosting App Service. On S1 that means that there are over ~2000 TCP connections opened by the webjobs in total.
All in all we can't explain why the listeners are so hungry for TCP connections. Each one is using one Topic-/QueueReceiver for their application lifetime and also a singleton HttpClient to connect to the target API. In theory that should mean each listener is never holding more than 10 TCP connections open at once.

I analyzed the code but found no reason for the high TCP connection demand.

All listeners rougly work like this (.NET Console Applications, hosted as continuous Azure Webjobs in App Services):

public static async Task Main(string[] args)
{
    var configuration = GetConfiguration();

    // Setup dependencies (e.g. Singleton HttpClient)
    IServiceCollection serviceCollection = new ServiceCollection();
    ConfigureServices(serviceCollection, configuration);

    IServiceProvider serviceProvider = serviceCollection.BuildServiceProvider();
    var factory = serviceProvider.GetService<TopicReceiverFactory<Model>>();
    var receiver = await factory.CreateAsync();
    receiver.ReceiveMessages();

    Console.ReadLine();
}

// ctor of the receiver used above
public QueueReceiver(QueueConfiguration configuration, IHandler<T> handler, ILogger<IReceiver> logger)
        : base(logger, handler)
{
    this.configuration = configuration;

    this.Client = new QueueClient(
    this.configuration.ConnectionString,
    this.configuration.QueueName,
    this.configuration.ReceiveMode);
}

// The ReceiveMessages Method used in Main
public void ReceiveMessages()
{
    var messageHandlerOptions = new MessageHandlerOptions(this.HandleExceptionReceivedAsync)
    {
        MaxConcurrentCalls = this.configuration.MaxConcurrentCalls,
        AutoComplete = false
    };

    this.Register(messageHandlerOptions);
}

protected void Register(MessageHandlerOptions messageHandlerOptions)
{
    if (messageHandlerOptions == null)
    {
        throw new ArgumentNullException(nameof(messageHandlerOptions));
    }

    this.Client.RegisterMessageHandler(this.ProcessMessageAsync, messageHandlerOptions);
}

The ProcessMessage roughly has this logic: Call the handler for the specific entity (posting the message to the api), if successful: Complete the message; if unsuccessful with critical Exceptions (eg JsonSerializerException because the format of the message is wrong) directly deadletter. Minor exceptions lead to the built in retry (up to ten times).

It's expected that the TCP connections are never exhausted. There isn't a lot happening in the environment.

EDIT: I found out that the outbound connection from the Listeners to the Service Bus is the source of the problem. The "TCP Connection" analyzer of the App Service shows this information:

Answer 1

We found the source of the problem. Consider the following architecture: A servicebus namespace with multiple topics and one queue. Messages are sent to the topics where Service Bus listeners are receiving and handling messages. If messages can't be handled they are forwarded to a central error handling queue. On this queue one listener is receiving the messages and reads the DeadLetterSource-Property on the message. In this property there is information about the originating topic.

Now here is the issue : Currently we are creating one TopicClient for each message. This happened because this listener should not need to know in advance which topics there are, thus reducing reusability. This however is not sustainable as we found out now, because you exhaust TCP connections.

The solution : We introduce the topic names via configuration so that this listener can create one TopicClient for each topic for the whole application lifestyle. Essentially there are n-Singleton TopicClient instances running at the same time.