故障转移后Redis主机后备

Question

We are trying to upgrade our redis/sentinel cluster from 2.8 to 3.2. Upgrade is going to happen under a heavy traffic. No downtime is acceptable for us.

Our setup has 6 redis/sentinels (3 sites, 2 servers on each site, each server has one instance of redis and sentinel running). Obviously we have 1 master and 5 slaves. We plan to upgrade our server one by one leaving our master server as the last one to upgrade.

By the time we are ready to upgrade our master, all slaves will be running on 3.2 and those slaves are able to sync up with master running on 2.8
Once we stop our master(2.8) or manually request a failover, a new server (3.2) will become master.

We want to keep our old master(2.8) around for a while, to be able to fallback in case if we find any issue with our new setup. Unfortunately an old redis(2.8) slave is not able to sync up with new master (3.2), because of different RDB format. We can stop our old server, slave(2.8), but we also want to be able to fallback to 2.8 as a master in case if we find an issue with 3.2. Since 2.8 is not able to sync up with 3.2, it would not have any data, thus can not be elected as new master by sentinel.

The question is how to fallback from 3.2 to 2.8 without losing any of the data?

Answer 1

I wouldn't recommend keeping a 2.8 node around as a fallback. Like you mentioned, you can't sync 3.2 -> 2.8 so falling back to 2.8 will mean all writes made to 3.2 will be lost.

I would recommend setting up a staging environment that runs 3.2 and do whatever testing you need to do there. Once you are feeling relatively confident about it, take a backup of your production database and go through the migration process.