C# SocketAsyncEventArgs and network glitches

Question

I've implemented async sockets using SAEA in the canonical way:

1. Have a pool of SAEAs ready for operations, keeping references so the GC doesn't have to move the memory
2. Replace the SAEAs when operations finish.

I've discovered an issue to do with network outages. Specifically, the sendAsync completion event does not fire when there's an outage , so my pool doesn't get refilled. This eventually ends up crashing the program, as it will run out of SAEAs, regardless of the size of the pool.

How do you avoid this? You can't simply keep a reference to an old SAEA, because that will give you an exception saying it's already waiting for an operation.

I'm looking for a way to gracefully replace the SAEAs. It's not a problem that the data does not get transmitted. One potential route to go is to run shutdown on the socket once a certain number of objects are waiting. Does anyone have experience of this?

Answer 1

It can indeed happen that no completion is posted in the case of basically infinite network latency (as would be the case when a cable is cut).

The only way to recover from such a problem is to implement some version of a heartbeating scheme. The SO_KEEPALIVE option exists on sockets but the default behavior is not usually what you want. From MSDN :

For TCP, the default keep-alive timeout is 2 hours and the keep-alive interval is 1 second.

I would suggest implementing your own heartbeating scheme, and consider the connection "dead" if no response heartbeat is received within a reasonnable amount of time. For example, you could :

1. Send "heartbeats" at regular intervals (or when no other message is being sent)
2. write down the time when heartbeats(or any) message is received, and set a timeout so that if the timeout is triggered then you have not received data for too long and the connection should be shutdown and re-initiated

Answer 2

The proposed solution does indeed work. It's simply a matter of checking before each send whether some limit is exceeded, and shutdown+close the socket if it is. This prevents the objects from leaking, allowing graceful reconnection later on.

I tested this by unplugging my cable and reconnecting repeatedly. In the old case the pool would empty. Now it survives the network outage.