SignlaR not working with load balancing

Question

We already had a server with signalr hub where clients connect using signalr. We recently moved it to two nodes with a load balancer for scalability. But after moving it to load balancer now clients are unable to connect to the server using signalr. Clients first try using web sockets and it gives the following error on signalr trace.

   fae26beb-a806-4957-b52a-39b80856e492 - Auto: Failed to connect to using transport webSockets. System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
   at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNet.SignalR.Client.Transports.WebSocketTransport.<PerformConnect>d__1.MoveNext()

But after couple of times trying to connect it manages to connect using SSE (Server Sent Events).

   fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BA9F","S":1,"M":[]})  
   fae26beb-a806-4957-b52a-39b80856e492 - ChangeState(Connecting, Connected)
   fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BAA0","M":[{"H":"PrintConnectorHub","M":"SignalRConnectedSuccessfully","A":[true]}]})  
   fae26beb-a806-4957-b52a-39b80856e492 - OnMessage({"R":false,"I":"3"}) 

But when the server tries to send a message back to the client, client does not receive the message. We have implemented SignalR scale out with SQL Server and following are the log traces in the server.

TRACE  SignalR.SqlMessageBus  Stream 0 : SqlReceiver last payload ID=47781, new payload ID=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord) 
TRACE  SignalR.SqlMessageBus  Stream 0 : Updated receive reader initial payload ID parameter=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord) 
TRACE  SignalR.SqlMessageBus  Stream 0 : Payload 47782 containing 1 message(s) received (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord) 
TRACE  SignalR.SqlMessageBus  Stream 0 : 1 records received (Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation.ExecuteReaderWithUpdates) 
TRACE  SignalR.SqlMessageBus  Created DbCommand: CommandType=Text, CommandText=SELECT [PayloadId], [Payload], [InsertedOn] FROM [SignalR].[Messages_0] WHERE [PayloadId] > @PayloadId, Parameters= [Name=PayloadId, Value=47782] (Microsoft.AspNet.SignalR.SqlServer.DbOperation.TraceCommand)

Signalr initialization in clients

signalrHostUrl = "SignalR_Host"; 

try
{
    hubConnection = new HubConnection(signalrHostUrl);
    hubConnection.Error += ex =>
    {
        signalRlogger.Error(ex, "Exception from hubConnection");
    };

    hubConnection.ConnectionSlow += () => ConnectionSlow();
    hubConnection.Closed += () => SignalrConnectionConnectionClosed();
    hubConnection.StateChanged += signalRConnectionStateController.StateChanged;

    hubConnection.Start().Wait();
}
catch (Exception ex)
{
    signalRlogger.Error(ex, "initialzing hub connection");
}

Before moving to the load balancer the clients were perfectly able to connect via web sockets and communicate. But after moving to load balancer it completely fails to connect via web sockets. We even tried to open a web socket connection by this tool ( https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en ) which also fails to open a connection. Although it manages to connect via SSE it still unable to communicate to clients. What are the complications when using web sockets in load balancing and in this situation why signalr fails to make a connection to the clients?

Answer 1

When the other machine tries to decrypt the connection token you got when negotiating the connection it can't and the request fails. You need to ensure that all machines you are using have the same machine key. This way all the machines will be able to decrypt connectionTokens created by other machines.

Are you able to enable stickyness on the load balancers? For example session based stickyness. This will make sure the clients connect to the same machine. The downside is that if a machine fails the client won't be able to reconnect.