PouchDB / CouchDB synchronization and database design upgrade tools andgood practices

Question

I am developing an ionic application using PouchDB to store data that I plan to sync with CouchDB and I am wondering what are the tools and/or good practices to manage database design upgrades.

By database design upgrade I mean upgrading the structure of the documents or the algorithm to calculate their ids (I am using DocUri and allDocs queries and may have to update the structure of the ids to meet new requirements) but that could also be indexes upgrades.

Without database synchronization I would just store a design version number in the database, check this version number at application start time and trigger upgrade actions when needed.

Adding database synchronization seems to make things much more complex. Even when using the "one database per user" paradigm, an user may have several instance of the application running in several browser instances or hybrid apps on several devices. If one of these instances performs a database upgrade that may render the data unusable for the other instances. If several instances start a database upgrade at the same time this might lead to conflicts and in the best case to a lot unnecessary traffic and duplication of work.

I think that some kind of orchestration is needed and the basics for a possible scenario could be:

• When an instance detects that an upgrade is needed, it starts by updating the database design version number with a flag to mention that an upgrade is happening.
• When the other instances receive the replication of this design version number they suspend the synchronization and inform their users that they could (or should) download a new version.
• When another instance requiring the same version number sees that an upgrade

Of course the devil is in the details and that raises more questions:

• How do other instances know that the upgrade is finished if they have suspended the synchronization? Do they need to poll the CouchDb server directly?
• How do we deal with upgrades terminated or paused by a user closing or pausing a device of by any kind of error?
• After the upgrade is considered as terminated by the instance which has initiated this upgrade, other instances will still have documents with the previous structure, how do they differentiate documents which have been upgraded and will be sync from CouchDB with documents which needs to be upgraded?
• ...

That seems complex and messy but everyone syncing data must have had the same issue...

How do you handle these upgrades in you applications?

Answer 1

First off, I'd make updates in the document structure as backwards compatible as possible:

• rely more on allDocs() and processing documents based on known properties (eg a type ) instead of its _id if that's subject to change from time to time.
• simply ignore documents of an unknown type or with unknown properties, don't treat them as errors.
• always introduce new documents and properties rather than trying to modify the existing ones.

It's also useful to have a versioning logic:

• lock out deprecated client versions and enforce that the app must be upgraded.
• if documents with a version above the version configured in the client are received, don't process them.

So if a schema upgrade is necessary, I'll usually:

1. Release a client version that supports both the old and the new document structure. Notify the user that he should upgrade.
2. Once that version has enough penetration, enforce it as required version to prevent modifications by clients with the old version.
3. Perform a server side migration, if needed.
4. Release a client version that supports only the new schema.

In general, I'd try to avoid modifying data directly in the client's PouchDB. Instead, the server should have the central authority and the PouchDBs are as read-only as possible.