TCP 连接在 ASP.NET + Azure 中停止

Question

I have a real head scratcher here (for me).

I have the following setup:

• Kube.netes Cluster in Azure (linux VMs)
• ASP.NET docker image with TCP server
• Software simulating TCP clients
• RabbitMQ for notifying incoming messages

Peer behaviour:

• The client sends its heartbeat every 10 minutes
• The server sends a keep-alive every 5 minutes (nginx-ingress kills connections after being idle for ~10 minutes)

I am testing the performance of my new TCP server. The previous one, written in Java, could easily handle the load I am about to explain. For some reason, the new TCP server, written in C#, loses the connection after about 10-15 minutes.

Here is what I do:

• Use the simulator to start 500 clients with a ramp-up of 300s
• All connections are there established correctly
• Most of the time, the first heartbeats and keep-alives are sent and received
• After 10+ minutes, I receive 0 bytes from Stream.EndRead() on BOTH ends of the connection.

This is the piece of code that is triggering the error.

var numberOfBytesRead = Stream.EndRead(result);
if (numberOfBytesRead == 0)
{
    This.Close("no bytes read").Sync(); //this is where I end up
    return;
}

In my logging on the server side, I see lots of disconnected ('no bytes read') lines and a lot of exceptions indicating that RabbitMQ is too busy: None of the specified endpoints were reachable .

My guesses would be that the Azure Load Balancer just bounces the connections, but that does not happen with the Java TCP server. Or that the ASP.NET environment is missing some configuration.

Does anyone know how this is happening, and more important, how to fix this?

--UPDATE #1--

I just used 250 devices and that worked perfectly.
I halved the ramp-up and that was a problem again. So this seems to be a performance issue. A component in my chain is too busy.

--UPDATE #2--

I disabled the publishing to RabbitMQ and it kept working now. Now I have to fix the RabbitMQ performance.

Answer 1

I ended up processing the incoming data in a new Task. This is my code now:

public void ReceiveAsyncLoop(IAsyncResult? result = null)
{
    try
    {
        if (result != null)
        {
            var numberOfBytesRead = Stream.EndRead(result);
            if (numberOfBytesRead == 0)
            {
                This.Close("no bytes read").Sync();
                return;
            }

            var newSegment = new ArraySegment<byte>(Buffer.Array!, Buffer.Offset, numberOfBytesRead);

            // This.OnDataReceived(newSegment)); <-- previously this
            Task.Run(() => This.OnDataReceived(newSegment));
        }

        Stream.BeginRead(Buffer.Array!, Buffer.Offset, Buffer.Count, ReadingClient.ReceiveAsyncLoop, null);
    }
    catch (ObjectDisposedException) { /*ILB*/ } 
    catch (Exception ex)
    {
        Log.Exception(ex, $"000001: {ex.Message}");
    }
}

Now, everything is super fast.