持续处理来自PostGre数据库的数据-采取什么方法？

Question

Have a question about what sort of approach to take on a process I am trying to structure. Working with PostgreSQL and Python.

Scenario:

• I have two databases A and B.
• B is a processed version of A.
• Data continually streams into A, which needs to be processed in a certain way (using multi-processing) and is then stored in B.
• Each new row in A needs to be processed only once.

So:

streamofdata ===> [database A] ----> process ----> [database B]

Database A is fairly large (40 GB) and growing. My question is regarding the determination on what is the new data not yet processed and put into B. What is the best way to determine what rows have to be processed still.

Matching primary keys each time on what has not yet been processed is not the way to go I am guessing

So let's say new rows 120 to 130 come into database A over some time period. So my last row processed row was 119. Is a correct approach to look at the last row id (the primary key) 119 processed and say that anything beyond that should now be processed?

Also wondering whether anyone has any further resources on this sort of 'realtime' processing of data. Not exactly sure what I am looking for technically speaking.

Answer 1

Well, there are a few ways you could handle this problem. As a reminder, the process you are describing is basically re-implementing a form of database replication, so you may want to familiarize yourself with the various popular replication options out there for Postgres and how they work, particularly Slony might be of interest to you. You didn't specify what sort of database "database B" is, so I'll assume it's a separate PostgreSQL instance, though that assumption won't change a whole lot about the decisions below other than ruling out some canned solutions like Slony.

1. Set up a FOR EACH ROW trigger on the important table(s) you have in database A which need to be replicated. Your trigger would take each new row INSERTed (and/or UPDATEd, DELETEd, if you need to catch those) in those tables and send them off to database B appropriately. You mentioned using Python, so just a reminder you can certainly write these trigger functions in PL/python if that makes life easy for you, ie you should hopefully be able to more-or-less easily tweak your existing code so that it runs inside the database as a PL/Python trigger function.

2. If you read up on Slony, you might have noticed that proposal #1 is very similar to how Slony works -- consider whether it would be easy or helpful for you to have Slony take over the replication of the necessary tables from database A to database B, then if you need to further move/transform the data into other tables inside database B, you might do that with triggers on those tables in database B.

3. Set up a trigger or RULE which will send out a NOTIFY with a payload indicating the row which has changed. Your code will will LISTEN for these notifications and know immediately which rows have changed. The psycopg2 adapter has good support for LISTEN and NOTIFY. NB you will need to exercise some care to handle the case that your listener code has crashed or gets disconnected from the database or otherwise misses some notifications.

4. In case you have control over the code streaming data into database A, you could have that code take over the job of replicating its new data into database B.