Application with auto update and MongoDB replica set

Question

I'm currently in a process of providing our applications of a high availability capability.

Our applications have an auto update feature (using Mongeez) that ensures that the structure of all databases/collections/documents are on correct format for that application version.

The problem is that I'm having a hard time, on how to resolve the replication and versioning problem.

I'll use the following example to better explain the problem.

I need to update Application X to Application X2.0 on both servers without downtime, so I do the following:

1. Stop Application X A
2. Install the 2.0 version on Server A
3. Launch Application X2.0 A
4. Application X2.0 A automatically updates MongoDB to the 2.0 format
5. Since we are using a MongoDB Replica Set, it automatically propagates the changes in all MongoDB instances
6. PROBLEM!!!... Application X B is still running version 1.0 but it's using the 2.0 version of the databases/collections/documents structure

How can I solve this issue? What am I doing wrong in the previous "workflow"?

Answer 1

Let's suppose:

• F1: Format 1.0
• F2: Format 2.0
• XA: reads & writes F1
• XB: new application, reads & writes F2 (document 2.0)

If F2 is compatible with F1, you can make XA & XB run together. It seems that this cause isn't what you described :)

If F2 is not compatible with F1, there is a lot of work to do.

=============== SIMPLE METHOD ==================

1. add a version field to your document (or use a field to distinguish between F1 and F2)
2. make you XB compatible with F1&F2: if it reads F1 then writes F1, if it reads F2 then writes F2
3. upgrade all applications to XB (XA is not affected because there are no F2 at this moment)
4. make XB only writes F2: if it reads F1 then writes F2, re-deploy XB.
5. convert all F1 documents to F2

=============== COMMON BUT COMPLEX METHOD ==================

Let's define some concepts first:

• C1: collection for F1
• C2: collection for F2
• XB: an application with a switch to control the reading: from C1 or C2. it uses XA's logic to read from C1.

Steps:

1. XA running, reads & writes F1 in C1
2. migrate all data from C1 to C2 (upgrade from F1 to F2)
3. analyze and execute oplog to make sure all new modifications in C1 migrated to C2
4. XB reads from C1, write to C1&C2 (IMPORTANT!)
5. deploy XB step by step, XA is not affected because C1 is usable
6. switch all XB to read from C2
7. remove legacy code for F1 in XB

You need some programs(scripts) to do step2 & step3.

If you don't care about cold data, you can skip step2&step3, let XB run for some time to migrate hot data.

=============== another method ==================

1. divide your server into to groups. eg: G1(XA1, XA2), G2(XA3, XA4)
2. use nginx/haproxy to balance your requests
3. make all requests sent to G1(XA1, XA2)
4. deploy XB to G2, result: G2(XB3, XB4)
5. make all requests sent to G2(XB3, XB4)
6. deploy XB to G1, result: G1(XB1, XB2)
7. make all requests sent to G1(XB1, XB2)/G2(XB3, XB4)
8. convert all F1 documents to F2

there are risks:

1. In step 3/step 5, your server load may exceed, so you need to do this in non-business time
2. If you have bugs in XB, there is no way to rollback.