How mongo db primary is selected when environment is restarted?

Question

I have set up an mongo cluster with 3 servers say A,B,C. My application writes lot of data db, hence oplog size is always very high.

When environment is restarted, Server A is primary ( we always run the rs.add(a), rs.add(B), rs.add(C) on A), then Server C becomes primary due to xyz reason - may be due to server reboot oo temporary connection loss. Then application has written data to Server C, and Server A is still trying to remain in sync with Server C.

Now environment is restarted, my understanding is that server C should be chosen as primary. However server A is being chosen as primary.

Could anyone please explain the reason why Server A is chosen as primary.

Answer 1

Assuming that you are using majority write concern for data that you actually care about, server C would not have acknowledge the writes from the application until at least one of the secondary nodes also wrote the data.

A 3-node replica set needs to have at least 2 nodes online in order to elect a primary, and the election process will favor the node(s) with the most recent timestamp.

If nodes A and B came online slightly faster than C, they might complete an election before C is available.

In this situation, if you were using majority write concern there would be no data loss as either A or B would also contain everything that had been written to and acknowledged by C.

However, if you were using write concern with w:1 or w:0 , there might be writes that C acknowledged that neither A nor B contain. When A and B come online, they would still elect a primary because neither knows anything about the additional data on C. When C then becomes available, if there have been any writes, C will have to conduct rollback in order to participate in the replica set.

As to why server A became primary in your cluster, check the mongod logs. In modern version of MongoDB election information is logged so you will be able to determine which nodes participated in the election, and which nodes voted for server A.

Answer 2

If you want a specific server to be preferred for the primary role, you can set member priorities accordingly. Absent priorities, MongoDB is free to have any of the members of the replica set as the primary.

Setting priorities does not guarantee that the server you want is ALWAYS going to be the primary - if this server gets restarted some other node will be a primary, and when this server comes back up it will not be eligible to be a primary again until it syncs with the current primary.