Mongodb architecture and failover with two data centres

Question

I'm trying to figure out whether there is a way to seamlessly failover a mongo replicaset where most of the mongodb nodes live in the primary data centre. My current limitation is 2 data centres and third datacentre is out of the question. The issue I have is that if data centre 1 goes down, the secondary node in data centre 2 will not be promoted to primary without manual intervention.

Data centre 1 (Primary): Mongo Node (Primary) Mongo Node (Arbiter)

Data centre 2 (Secondary): Mongo Node (Secondary)

I've looked at mongodb whitepapers but they state manual intervention is required to make the mongodb instance in dc2 primary if dc1 is lost.

My question is whether there is an architecture out there or configuration that will make it possible to lose data centre 1 and still have the ability to have a data centre 2 takeover with write enabled without manual intervention/reconfiguration. Is this possible without going down a 3 data centre architecture path. Is it possible to keep two 3 member replica sets at each site synchronised and potentially do the failover at a network level for the connecting applications?

Thanks.

Answer 1

If you go with 2 data centers to me easiest solution is to cover only fail in Primary. Good news is if Slave is dead - you only need to wait.

If access to Primary fails you need to callback procedure that will force Slave to Primary. This switch will cause downtime in your application if you don't spent more time to create a gateway that will buffer queries and waits for callback from the switch. In that way you will have only slowness with increase timeout.

After Primary is live again you need to connect back to it (because your Slave node is not reliable) - this will cause again downtime- you need another process that checks if Primary is alive (from data center 2) and if it is trigger event and proceed with callback.

Manual intervention to force Slave as Primary can be wrapped to script.

To me here best solution is to go with 3rd data center where arbiter will stay. The effort to skip that and put application logic there is not worthy. Automatic failover in Mongo works very well and its reliable. You may have lots of problems if you go with application logic to achieve that with 2 data centers ... I rather go with their recommendation.

Answer 2

First, as you have noticed, you cannot do automatic fail over with only two nodes. Second, money is not real issue when you think that "third" data center. You may ask why or "how so"? You need arbiter, as you know. Arbiter don't need resources really, any small Linux machine will do fine. Small VPS machines don't cost that much. Here you can find machine 1 x 2.40 GHz, 512 MB, 20 GB with only 1,24€/month. From here you get beefier machine with 1.99€/month.

Actually both those places could run quite big mongodb with those "tiny" machines.