Concurrent Clients issue in C# Sockets

Question

I am working on a client server application, Windows Server and Linux Client. I was testing my server with multiple concurrent clients. I tried just 20 concurrent connections from client, and i noticed that some requests were not processed despite all 20 requests were the same. They went into the queue and for some reason when their turn comes client was shutdown (Client connect timeout is 5 sec).

Then I added a Thread.Sleep(1000), to check if it is really asynchronous but then i realized it does not process other request until timeout. Despite the fact

1. It is asynchronous
2. ManualResetEvent was set before going to sleep.

Now I am wondering what Am I missing here, as this happens with concurrent connections mostly?

public static void StartServer(IPAddress ipAddr, int port)
{
    //IPEndPoint serverEndPoint = new IPEndPoint(ipAddr, port);
    IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Any, port);
    Socket clientListener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    try
    {
        clientListener.Bind(serverEndPoint);
        clientListener.Listen(500);
        Console.WriteLine("-- Server Listening: {0}:{1}",ipAddr,port);
        while (true)
        {
            resetEvent.Reset();
            Console.WriteLine("|| Waiting for connection");
            clientListener.BeginAccept(new AsyncCallback(AcceptConnection), clientListener);
            resetEvent.WaitOne();
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
}


public static void AcceptConnection(IAsyncResult ar)
{
    // Get the socket that handles the client request.
    Socket listener = (Socket)ar.AsyncState;
    Socket handler = listener.EndAccept(ar);
    // Signal the main thread to continue.
    resetEvent.Set();
    // Create the state object.
    JSStateObject state = new JSStateObject();
    state.workSocket = handler;
    if (handler.Connected)
    {
        Console.WriteLine("** Connected to: {0}", handler.RemoteEndPoint.ToString());
        state.workingDirectory = JSUtilityClass.CreatetTemporaryDirectry();
        try
        {
            Thread.Sleep(1000);
            Receive(state);
        }
        catch (Exception e)
        {
            handler.Shutdown(SocketShutdown.Both);
            handler.Close();
            Console.WriteLine(e.Message);
        }
    }
}

Answer 1

I created a test that sends 100 connection attempts and found a few things slowing it down.

Why is it so slow?

I put a breakpoint in AcceptConnection to look at the callstack, this is it

ConsoleApplication1.exe!ConsoleApplication1.Program.AcceptConnection(System.IAsyncResult ar) Line 62    C#
        System.dll!System.Net.LazyAsyncResult.Complete(System.IntPtr userToken) + 0x69 bytes    
        System.dll!System.Net.ContextAwareResult.CaptureOrComplete(ref System.Threading.ExecutionContext cachedContext, bool returnContext) + 0xab bytes    
        System.dll!System.Net.ContextAwareResult.FinishPostingAsyncOp(ref System.Net.CallbackClosure closure) + 0x3c bytes  
        System.dll!System.Net.Sockets.Socket.BeginAccept(System.AsyncCallback callback, object state) + 0xe3 bytes  
        ConsoleApplication1.exe!ConsoleApplication1.Program.StartServer(System.Net.IPAddress ipAddr, int port) Line 48 + 0x32 bytes C#

So the callback AcceptConnection is running from the same thread that BeginAccept was called from. I had a look at FinishPostingAsyncOp with reflector and it's using the async pattern where if there is already a socket operation in the queue waiting to be processed, it'll do so on the current thread, otherwise if there isn't anything pending, it'll process in a different thread later on, eg

SocketAsyncEventArgs sae = new SocketAsyncEventArgs();
sae.Completed += new EventHandler<SocketAsyncEventArgs>(SocketOperation_Completed);
if (!clientListener.AcceptAsync(sae))
    AcceptConnection(clientListener, sae); // operation completed synchronously, process the result
else
    // operation will complete on a IO completion port (different thread) which we'll handle in the Completed event

So as you observed the program is effectively completely synchronous in this scenario, and with the 1 second Thread.Sleep it's going to take at least 100 seconds to accept all the connections, by which time most of them will timeout.

The solution

Even though BeginAccept method summary says

Begins an asynchronous operation to accept an incoming connection attempt.

It turns out there is more to the story

From MSDN http://msdn.microsoft.com/en-AU/library/system.net.sockets.socket.beginaccept.aspx

BeginAccept(Int32, AsyncCallback, Object) Begins an asynchronous operation to accept an incoming connection attempt and receives the first block of data sent by the client application.

So it's performing a read operation with a short timeout before firing the callback. You can disable this by specifying the receiveSize of 0. Change

clientListener.BeginAccept(new AsyncCallback(AcceptConnection), clientListener);

to

clientListener.BeginAccept(0, new AsyncCallback(AcceptConnection), clientListener);

That speeds it up, and if we remove the Thread.Sleep(1000) from AcceptConnection then all the connections are accepted really fast.

If you leave the Thread.Sleep(1000) in there to simulate work load or just for testing then you may want to prepare the server to handle such a load by doing

int minWorkerThreads = 0;
int minCompletionPortThreads = 0;
ThreadPool.GetMinThreads(out minWorkerThreads, out minCompletionPortThreads);
ThreadPool.SetMinThreads(minWorkerThreads, 100);

Where 100 is the amount of threads you want readily available to handle socket operations.

Just one other thing, it's a matter of personal preference but just so you know you might like to call BeginAccept from within AcceptConnection which removes the need for that while loop. ie change this

while (true)
{
    resetEvent.Reset();
    Console.WriteLine("|| Waiting for connection");
    clientListener.BeginAccept(new AsyncCallback(AcceptConnection), clientListener);
    resetEvent.WaitOne();
}

to this

Console.WriteLine("|| Waiting for connection");
clientListener.BeginAccept(new AsyncCallback(AcceptConnection), clientListener);

and put another BeginAccept in AcceptConnection

public static void AcceptConnection(IAsyncResult ar)
{
    // Get the socket that handles the client request.
    Socket listener = (Socket)ar.AsyncState;
    // start another listening operation
    listener.BeginAccept(new AsyncCallback(AcceptConnection), listener);
    ... the rest of the method
}