Remoting server forcibly closing client connections in the middle of remote calls

Question

I have a system consisting of a server accepting remoting calls from clients with TCP as the underlying transportlayer. It normally works like a charm, but if I increase the no. of clients, the server starts at random to close the TCP connections in the middle of the calls. Not all calls are interrupted this way.

That is really unexpected behaviour... I get no exceptions on the server side, just the client side exception:

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Server stack trace: 
   ved System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   ved System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   ved System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
   ved System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
   ved System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
   ved System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   ved System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   ved System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
   ved System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   ved System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   ved System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   ved System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   ved EBH.GuG.AgentKit.Transports.RemotingAgentHostEndPoint.SyncInvoke(Agent a, Int32 port)

Answer 1

Are you running on windows XP/2000/98 ?

If so, XP has an inbuilt throttling mechanism of 10 outbound sockets (to stop you using desktop machines as servers to force you to pay for windows Server) My hunch is that you may be hitting this limit.

Additional:

Perhaps you could rearchitect your calls with a callback so that they don't maintain open sockets while work is being performed, which should improve your concurrent throughput.

Answer 2

How well do you know the network hardware between your clients and your server?

Every time I've had this kind of problem, it's always turned out to be caused by a misconfigured firewall, load balancer etc.

If you set up a test environment with your clients and server connected to the same switch, you should be able to perform a load test to work out if your code fails without any other network hardware involved...