Redis failover in a K8s cluster

Question

I am trying to get Redis failover to work in Kubernetes with a worker-node failure scenario. I have a K8s cluster that consists of a master node and two worker nodes. The master node does not schedule pods. The manifests for Redis are such that there is a master and a slave instance in a stateful set and 3 sentinels in another stateful set. The manifests have affinity to steer the pods to be scheduled on separate worker nodes. If I drain a worker node that has the master instance and one sentinel, failover works like a champ.

If, however, there are 2 sentinels that are evicted with the master instance, no master is elected and the 2 sentinels that are re-started on the remaining worker node report: -failover-abort-no-good-slave master jnpr-ipb-redis-masters 10.244.1.209 7380 . That IP address in the log message is the IP address of the former slave (which I expected to be promoted to the new master).

Is there a bit of wizardry to make this work? Is this a valid cluster config? Not really sure what I should be looking at to get an idea of what is happening.

Answer 1

What you want is a PodDisruptionBudget. That will make voluntary evictions at least not break things. Beyond that you can use hard anti-affinities to force the pods to be scheduled on different nodes. Failures are still possible though, if you lose two nodes at the same time the Sentinels can desync. This is a large part of why Redis Sentinel is mostly no longer used in favor of Cluster Mode.