Using Azure load balancer to reboot/update server with zero downtime

Question

I have a really simple setup: An azure load balancer for http(s) traffic, two application servers running windows and one database, which also contains session data.

The goal is being able to reboot or update the software on the servers, without a single request being dropped . The problem is that the health probe will do a test every 5 seconds and needs to fail 2 times in a row. This means when I kill the application server, a lot of requests during those 10 seconds will time out. How can I avoid this?

I have already tried running the health probe on a different port, then denying all traffic to the different port, using windows firewall. Load balancer will think the application is down on that node, and therefore no longer send new traffic to that specific node. However... Azure LB does hash-based load balancing. So the traffic which was already going to the now killed node, will keep going there for a few seconds!

Answer 1

First of all, could you give us additional details: is your database load balanced as well ? Are you performing read and write on this database or only read ?

For your information, you have the possibility to change Azure Load Balancer distribution mode, please refer to this article for details: https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-distribution-mode

I would suggest you to disable the server you are updating at load balancer level. Wait a couple of minutes (depending of your application) before starting your updates. This should "purge" your endpoint. When update is done, update your load balancer again and put back the server in it. Cloud concept is infrastructure as code: this could be easily scripted and included in you deployment / update procedure.

Another solution would be to use Traffic Manager. It could give you additional option to manage your endpoints (It might be a bit oversized for 2 VM / endpoints).

Last solution is to migrate to a PaaS solution where all this kind of features are already available (Deployment Slot).

Hoping this will help.

Best regards