Mutex implementation within IIS async controller action?

Question

I have written a web service that's an Asp.Net MVC application hosted in IIS. The data from the web service isn't retrieved from a database but from another server that is accessed via TCP. We'll call this the Data Server. There can be multiple Data Servers that the web service connects to.

For the sake of discussion, a user authenticates by specifying a username and password then a "Session" with a Socket is created with the appropriate Data Server. Assuming everything is good - we keep the Socket alive and pass the user a token that identifies the Session they belong to.

For each Socket I need to prevent traffic from interrupting each other. I assume that the best way to do that is to run the network traffic on a Socket in a serialized manner. I have achieved this by using a lock. My code to execute a query on this Data Server follows. The problem I'm having is that once in a while it appears that I'm getting two queries that collide and one may hang. I see a query come in but it looks like it gets stuck at the lock. Is the lock mechanism safe for IIS async calls? Is there instrumentation I can put in to make sure this is actually the bug? I've been looking for a while but I can't find guidance for this particular scenario.

 private async Task<string> query(string request, AbstractPermission context = null, bool bUpdateDateTime = true)
        {
            try
            {
                string reply = "";
                isErrorMsg = false;

                //this lock prevents the keep alive thread from coming in here while we're on the socket
                lock (socketLock)
                {
                    sendDone.Reset();
                    receiveDone.Reset();

                    // Send test data to the remote device.  
                    Send(Socket, request);
                    sendDone.WaitOne();

                    // Receive the response from the remote device.
                    Receive(Socket);
                    receiveDone.WaitOne();

                    reply = QueryResponse;

                }  //done reading - let's unlock

                if (bUpdateDateTime)
                {
                    this.LastUsed = DateTime.Now;
                }

                return QueryResponse;
            }
            catch (Exception e)
            {
                throw e;
            }
        }


  private void Send(Socket client, String data)
        {
            byte[] byteData = Encoding.ASCII.GetBytes(data);

            client.BeginSend(byteData, 0, byteData.Length, 0,
                new AsyncCallback(SendCallback), client);
        }

        private void SendCallback(IAsyncResult ar)
        {
            try
            {
                // Retrieve the socket from the state object.  
                Socket client = (Socket)ar.AsyncState;

                // Complete sending the data to the remote device.  
                int bytesSent = client.EndSend(ar);

                // Signal that all bytes have been sent.  
                sendDone.Set();
            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }
        }


        private void Receive(Socket client)
        {
            try
            {
                // Create the state object.  
                StateObject state = new StateObject();
                state.workSocket = client;
                state.PostInitialRead = false;

                // Begin receiving the data from the remote device.  
                client.BeginReceive(state.buffer, 0, StateObject.BufferSize, SocketFlags.None,
                    new AsyncCallback(ReceiveCallback), state);
            }
            catch (Exception e)
            {
                LogError(e);
            }
        }

        private void ReceiveCallback(IAsyncResult ar)
        {
            try
            {
                // Retrieve the state object and the client socket   
                // from the asynchronous state object.  
                StateObject state = (StateObject)ar.AsyncState;
                Socket client = state.workSocket;
                bool PostInitialRead = state.PostInitialRead;

                // Read data from the remote device.  
                int bytesRead = client.EndReceive(ar);

                //
                //
                var thisBatch = Encoding.ASCII.GetString(state.buffer, 0, bytesRead);
                var endIdx = thisBatch.IndexOf('\x04');

                if (!PostInitialRead)
                {
                    if (bytesRead == 0)
                        throw new ApplicationException("Timeout waiting for response");

                    if (endIdx != -1)
                    {
                        thisBatch = thisBatch.Substring(0, endIdx);
                    }
                    if (state.buffer[0] != 0)
                    {
                        thisBatch = thisBatch.Substring(1, state.buffer[0]);
                        isErrorMsg = true;
                    }
                    else if (state.buffer[1] != 0)
                    {
                        thisBatch = thisBatch.Substring(2);
                        isErrorMsg = true;
                    }
                    else
                    {
                        thisBatch = thisBatch.Substring(2);
                    }
                    state.sb.Append(thisBatch);
                    state.PostInitialRead = true;
                }
                else
                {
                    state.ms.Write(state.buffer, 0, endIdx!=-1?endIdx:bytesRead);
                }

                if (endIdx != -1)
                {
                    // Got everything
                    state.sb.Append(Encoding.ASCII.GetString(state.ms.ToArray()));
                    QueryResponse = state.sb.ToString();
                    receiveDone.Set();
                    return;
                }
                else
                {
                    // Get the rest of the data.  
                    client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                        new AsyncCallback(ReceiveCallback), state);
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }
        }

Answer 1

1: Is hosting something like this within IIS completely absurd?

No. See, dotnetcore and modern development uses SignalR based on WebSockets for stuff like that - and it is hosted in IIS, so it is NOT completely absurd.

2: We can not really answer that. The IO part is trivial - IIS can handle this. But without details on the data server - no idea. YOu generally want to avoid locks as much as possible, but that does not mean totally. MRSV (multiple Reader, Single Writer), copy on write etc. can help minimizing writes, but at the end you will need SOME locking. It is NOT fun debugging this, but that is what people doing that get paid big bucks for.

General applicatiosn avoid all that by offloading the locking to the database at the backend - which spends a LOT of time by many people optimizing locking on data. If you can not do that (remember, we do not know at all what your data server does internally) - welcome to the hard programming.

Your best chance for debugging is trying to find a repro case and then - while it is stuck - attach a debugger. Stuff like that is NOTORIOUSLY hard to debug - but again, this is like the topmost level of regular programming (leaving out certain hardware and very special tasks).