Synchronizing MongoDB server data to an IndexedDB local store

Question

I'm trying to evaluate using IndexedDB to solve the offline issue. It would be populated with data currently stored in a MongoDB database (as is).

Once data is stored in IndexedDB, it may be changed on the MongoDB server and I need to propagate those changes. Is there any existing framework or Library to do somehting like this for Mongo . I already know about CouchDB/PouchDB and am not exploring those two.

Answer 1

[ Sync solution for 2021 ]

I know the question asked was for MongoDB specifically, but since this is an old thread I thought readers might be looking for other solutions for new apps or rebuilds. I can really recommend to check out AceBase because it does exactly what you were looking for back then.

AceBase is a free and open source realtime database that enables easy storage and synchronization between browser and server databases. It uses IndexedDB in the browser, its own binary db / SQL Server / SQLite storage on the server. Offline edits are synced upon reconnect and clients are notified of remote database changes in realtime through a websocket (FAST!).

On top of this, AceBase has a unique feature called "live data proxies" that allow you to have all changes to in-memory objects to be persisted and synced to local and server databases, and remote changes to automatically update your in-memory objects. This means you can forget about database coding altogether, and code as if you're only using local objects. No matter whether you're online or offline.

The following example shows how to create a local IndexedDB database in the browser, how to connect to a remote database server that syncs with the local database, and how to create a live data proxy that eliminates further database coding. AceBase supports authentication and authorization as well, but I left it out for simplicity.

const { AceBaseClient } = require('acebase-client');
const { AceBase } = require('acebase');

// Create local database with IndexedDB storage:
const cacheDb = AceBase.WithIndexedDB('mydb-local');

// Connect to server database, use local db for offline storage:
const db = new AceBaseClient({ dbname: 'mydb', host: 'db.myproject.com', port: 443, https: true, cache: { db: cacheDb } });

// Wait for remote database to be connected, or ready to use when offline:
db.ready(async () => {

    // Create live data proxy for a chat:
    const emptyChat = { title: 'New chat', messages: {} };
    const proxy = await db.ref('chats/chatid1').proxy(emptyChat);  // Use emptyChat if chat node doesn't exist

    // Get object reference containing live data:
    const chat = proxy.value;

    // Update chat's properties to save to local database, 
    // sync to server AND all other clients monitoring this chat in realtime:
    chat.title = `Changing the title`;
    chat.messages.push({ 
        from: 'ewout', 
        sent: new Date(),
        text: `Sending a message that is stored in the database and synced automatically was never this easy!` +
              `This message might have been sent while we were offline. Who knows!`
    });

    // To monitor and handle realtime changes to the chat:
    chat.onChanged((val, prev, isRemoteChange, context) => {
        if (val.title !== prev.title) { 
            alert(`Chat title changed to ${val.title} by ${isRemoteChange ? 'us' : 'someone else'}`); 
        }
    });
});

For more examples and documentation, see AceBase realtime database engine at npmjs.com

Answer 2

Open up a changeStream with the resumeToken. There's no guarantee of causal consistency however since we're talking multiple disparate databases.

Answer 3

I haven't worked with IndexDB, but the design problem isn't that uncommon. My understanding of your app is that when the client makes the connection to MongoDB, you pull a set of documents down for local storage and disconnect. The client then can do things locally (not connected to the data server), and then push up the changes.

The way I see it you've got to handle two general cases:

1. when the MongoDB server is updated and breaks continuity with the client, the client will have to
   1. poll for the data (timer?) or
   2. keep a websocket open to let notifications free-flow over the pipe
2. when the user needs to push changed data back up the pipe
   1. you can reconnect asynchronously, check for state changes, (resolving conflicts according to your business rules)
   2. have a server side (light) interface for handling conflicts (depending on complexity of your app, comparing time stamps of state changes in MongoDB to IndexedDB updates should suffice)