在异步/等待TCP服务器中套接字断开连接后的大量时间延迟

Question

I've been upgrading some older software from the Begin/End pattern in C# to use the new async functionality of the TcpClient class.

Long story short, this receive method works great for small numbers of connected sockets, and continues to work great for 10,000+ connections. The problem comes when these sockets disconnect.

The method I am using server side is, in essence, this (heavily simplified but still causes the problem):

private async void ReceiveDataUntilStopped(object state)
        {
            while (IsConnected)
            {
                try
                {
                    byte[] data = new byte[8192];
                    int recvCount = await _stream.ReadAsync(data, 0, data.Length);
                    if (recvCount == 0) { throw new Exception(); }
                    Array.Resize(ref data, recvCount);
                    Console.WriteLine(">>{0}<<", Encoding.UTF8.GetString(data));
                }
                catch { Shutdown(); return; }
            }
        }

This method is called using ThreadPool.QueueUserWorkItem(ReceiveDataUntilStopped); when the connection is accepted.

To test the server, I connect 1,000 sockets. The time it takes to accept these is neglible, around 2 seconds or so. I'm very pleased with this. However, when I disconnect these 1,000 sockets, the process takes a substantial amount of time, 15 or more seconds, to handle the closure of these sockets (the Shutdown method). During this time, my server refuses any more connections. I emptied the contents of the Shutdown method to see if there was something in there blocking, but the delay remains the same.

Am I being stupid and doing something I shouldn't? I'm relatively new to the async/await pattern, but enjoying it so far.

Is this unavoidable behaviour? I understand it's unlikely in production that 1,000 sockets will disconnect at the same time, but I'd like to be able to handle a scenario like this without causing a denial of service. It strikes me as odd that the listener stops accepting new sockets, but I expect this is because all the ThreadPool threads are busy shutting down the disconnected sockets?

EDIT: While I agree that throwing an exception when 0 bytes are received is not good control flow, this is not the source of the problem. The problem is still present with simply if (recvCount == 0) { Shutdown(); return; } if (recvCount == 0) { Shutdown(); return; } if (recvCount == 0) { Shutdown(); return; } . This is because ReadAsync throws an IOException if the other side disconnects uncleanly. I'm also aware that I'm not handling the buffers properly etc. this is just an example, with minimal content, just like SO likes. I use the following code to accept clients:

private async void AcceptClientsUntilStopped()
        {
            while (IsListening)
            {
                try
                {
                    ServerConnection newConnection = new ServerConnection(await _listener.AcceptTcpClientAsync());
                    lock (_connections) { _connections.Add(newConnection); }
                    Console.WriteLine(_connections.Count);
                }
                catch { Stop(); }
            }
        }

Answer 1

if (recvCount == 0) { throw new Exception(); }

In case of disconnect you throw an exception. Exceptions are very expensive. I benchmarked them once at 10000/sec. This is very slow.

Under the debugger, exceptions are vastly slower again (maybe 100x).

This is a misuse of exceptions for control flow. From a code quality standpoint this is really bad. Your exception handling also is really bad because it catches too much. You meant to catch socket problems but you're also swallowing all possible bugs such as NRE.

        using (mySocket) { //whatever you are using, maybe a TcpClient
        while (true)
        {
                byte[] data = new byte[8192];
                int recvCount = await _stream.ReadAsync(data, 0, data.Length);
                if (recvCount == 0) break;
                Array.Resize(ref data, recvCount);
                Console.WriteLine(">>{0}<<", Encoding.UTF8.GetString(data));
        }
        Shutdown();
        }

Much better, wow.

Further issues: Inefficient buffer handling, broken UTF8 decoding (can't split UTF8 at any byte position!), usage of async void (probably, you should use Task.Run to initiate this method, or simply call it and discard the result task).

---

In the comments we discovered that the following works:

Start a high-prio thread and accept synchronously on that (no await). That should keep the accepting going. Fixing the exceptions is not going to be 100% possible, but: await increases the cost of exceptions because it rethrows them. It uses ExceptionDispatchInfo for that which holds a process-global lock while doing that. Might be part of your scalability problems. You could improve perf by doing await readTask.ContinueWith(_ => { }) . That way await will never throw.

Answer 2

Based on the code provided and my initial understanding of the problem. I think that there are several things that you should do to address this issue.

1. Use async Task instead of async void . This will ensure that the async state machine knows how to actually maintain its state.
2. Instead of invoking ThreadPool.QueueUserWorkItem(ReceiveDataUntilStopped); call ReceiveDataUntilStopped via await ReceiveDataUntilStopped in the context of an async Task method.

With async await , the Task and Task<T> objects represent the asynchronous operation. If you are concerned that the results of the await are executed on the original calling thread you could use .ConfigureAwait(false) to prevent capturing the current synchronization context. This is explained very well here and here too .

Additionally, look at how a similar "read-while" was written with this example .