Terminating WebSocket connections on AWS EBS behind ALB

Question

Does anyone have an approach to gracefully terminate WebSocket connections to Elastic Beanstalk behind an Application Load Balancer?

During scaling in, EC2 instances are terminated and immediately drop their connections. I'm wondering if there's a good way to handle the termination, or if I can only terminate an instance that no longer has any active connections.

Each WebSocket client is stateful, through the magic of sticky sessions/cookies. Reconnecting a client to a different issue is fine, but I want to avoid doing this when possible.

Answer 1

EC2 instances are terminated and immediately drop their connections. I'm wondering if there's a good way to handle the termination,

You can make the experience more graceful for your users:

Simple Approach:

There is a feature called 'Connection Draining', when an instance needs to shut down due to a scale-in event, it will wait for the predefined time for existing coonections to complete, it will not send new requests to that instance and after the specified duration it will forcefully terminate the remaining connections if any. So the time can be set longer than the Sticky session cookie time duration you specified, and also the longer than the TTL for WebSocket you have setup from your application and ensure that app retries to connect after this duration.

As you are using Elastic Beanstalk. The feature is under ELB -> Application -> Environment -> Configuration -> LoadBalancers

It should satisfy your current need.

More Dynamic (Complex) approach:

"graceful way to terminate those open connections."

"graceful" termination can happen when both the client and server are notified of the impending termination of the connection due to scale-in event so that they can then perform house-cleaning tasks before the connection is terminated ie instance being shut down.

It depends on your use-case if you want to venture into that route. But, if you need that, you could do the following

1. When the WebSocket connection is established. The client application is provided with the underlying EC2 instance id. (the counterpart of the websocket connection)

2. Create a cloud watch alarm for the cloud Watch event "Pending:Wait", the event details are here

3. Create an SNS topic for the cloud watch alarm notifications. The notification should have the ec2 instance id.

4. Both the client application and server applications subscribe to this topic. And when a notification arrives, they can check the ec2 instance id to verify if it is the host of the WebSocket connection. If yes, they prepare for the connection to be terminated on scale-in event.

As you can see, there are more moving part. But, if it improves the user experience and use case justifies it. Then it would be a solution.