Redis哨兵故障转移后返回旧主机

Question

I have 3 box setup of redis sentinel:

 CLIENT (connects to S1)
          |
          ↓
       +----+
       | M1 | us-east-1
       | S1 |
       +----+
          |
+----+    |    +----+
| R2 |----+----| R3 |
| S2 |         | S3 |
+----+         +----+
us-east-2      us-west-2

M1 - Master
S1 - Sentinel 1
S2 - Sentinel 2
S3 - Sentinel 3
R2 - First slave (R=replica)
R3 - Second slave

After my master died, sentinel made a failover to R2. I brought back M1 online (cleared some disk space) and now M1 is alive and well but a slave of R2. Is there an automatic way (or semi-automatic) to make M1 a master again and R2 as a slave of M1 and my traffic again using M1 as a master redis instance?

Essentially I want to revert to how it was prior to failover.

What currently happens is that it elects R2 as a master and reconfigures it to be:

CLIENT (connects to S1)
          |
          ↓
       +----+
       |[R2]| us-east-2
       | S2 |
       +----+
          |
+----+    |    +----+
|[M1]|----+----| R3 |
| S1 |         | S3 |
+----+         +----+
us-east-1      us-west-2

When I failover manually, it promotes R3 as master. (which is kind of expected).

But then when I failover again manually, it promotes R2, but I would expect it to promote M1.

All successive failovers rotate between R2 and R2 (while always keeping M1 as a slave of either).

My M1 slave priority is unspecified, so it means it's a default value of 100. My R2 slave priority is 200 and R2 is 300. That leads me to think that it should rotate all 3 of the boxes, but it rotates only R2 and R3 after the initial failover.

This looks like a sentinel bug to me

Answer 1

I am not sure why you want to do that in first place. Redis failing over to R2 and using at as master now should perfectly work as normal M1 instance. If that's not the case, you are not actually using Sentinel correctly for high availability.

You can just trigger a manual failover with SENTINEL failover R2 . It should switch to either M1 or R3.

Answer 2

I think kiddorails's answer is correct, but most probably you have a similar problem as I had, where for some reason your original master is not replicating correctly. Once I fixed my replication issue, I could cycle through my masters by issueing SENTINEL FAILOVER mymaster . Initially it would just bounce between the two original slaves, but now that my original master is correctly replicating, it is cycling through all 3. So I would recommend checking the replication of your original master after a failover. if you are sure it is working, you could also stop the other slave and then use the SENTINEL FAILOVER mymaster command to force a failover to the original master. If that fails, you know there must be an issue with the replication.