Real-Time Database Messaging

Question

We've got an application in Django running against a PGSQL database. One of the functions we've grown to support is real-time messaging to our UI when data is updated in the backend DB.

So... for example we show the contents of a customer table in our UI, as records are added/removed/updated from the backend customer DB table we echo those updates to our UI in real-time via some redis/socket.io/node.js magic.

Currently we've rolled our own solution for this entire thing using overloaded save() methods on the Django table models. That actually works pretty well for our current functions but as tables continue to grow into GB's of data, it is starting to slow down on some larger tables as our engine digs through the current 'subscribed' UI's and messages out appropriately which updates are needed as which clients.

Curious what other options might exist here. I believe MongoDB and other no-sql type engines support some constructs like this out of the box but I'm not finding an exact hit when Googling for better solutions.

Answer 1

Currently we've rolled our own solution for this entire thing using overloaded save() methods on the Django table models.

Instead of working on the app level you might want to work on the lower, database level.

Add a PostgreSQL trigger after row insertion, and use pg_notify to notify external apps of the change.

Then in NodeJS:

var PGPubsub = require('pg-pubsub');

var pubsubInstance = new PGPubsub('postgres://username@localhost/tablename');

pubsubInstance.addChannel('channelName', function (channelPayload) {
  // Handle the notification and its payload
  // If the payload was JSON it has already been parsed for you
});

See that and that .

And you will be able to to the same in Python https://pypi.python.org/pypi/pgpubsub/0.0.2 .

Finally, you might want to use data-partitioning in PostgreSQL. Long story short, PostgreSQL has already everything you need :)