如何设计一个pub-sub系统，同一实体可以有多个发布者？

Question

Consider the following situation:

1. There is a User table with fields like name, email, contact_no etc.
2. We have multiple products/system(with their own db) which use User information for various purpose.
3. These systems remain in sync by pub-sub pattern eg: System 1 changes name is consumed by system 2 and make changes in system 2.
4. For simplicity let us assume there are 3 systems:
   • S1 having UI for the user. Here user can himself change his information.
   • S2 system given to the sales person. Here user can call sales person and update their information.
   • S3 another product which uses information from S1 and S2 for various computations.

So information can be published from S1 and S2.

Suppose a user initially have name N1.

• At time t1, user updates name from N1 to N2.

• At time t1, sales person updates name from N1 to N3.

• Now S1 consumes event from S3 and updates name to N3. S2 consumes event from S1 and updates name to N2.

• In S3 name can be anything N2 OR N3 depending which event is consumed first.

This has caused a lot of data consistency among various system.

In ideal system there is only one publisher but due to requirement we had to add publishing events from Sales panel. What can be done to minimize data inconsitency?

Answer 1

The problem seems to be in deciding who is the master of User data, because right now S3 is trying to serve two masters.

S1 --> S3
S2 --> S3

Let's make S1 the master of User data, so that any other system which accepts that data (eg S2) is responsible only for updating the master. This way S3 gets User data from a single source (the master of User data).

       S1 --> S3
S2 --> S1 --> S3

The master is responsible for both consuming and producing the data that it owns. Another system may consume (and even store) the same data, but it can only produce to the master . No system can produce data that it doesn't own to any system other than the master of that data.

It doesn't really matter which system is the master, as long as there is only one. It doesn't really matter how many systems store copies of the data, as long as there is only one master. Having each system master one type of data is probably more scalable than a single master for all data.

Answer 2

Let me rephrase it once again..

1. you have multiple system which has independent DB attached to it.
2. User able to change data from any system
3. User should see data changed from any system.

In this scenario i would go with Master DB / System which will be single point of data source for every system.

If any system want to change data then data 1st need to update on Master Db / system then it should propagate to other system to reflect the changes.

for Pub-Sub I will follow Fan-IN and Fan-out method.

1. Every system will act as publisher as well as consumer for different topic / channel.
2. Publishers (which is on S1-SN) will push changes into one topic / channel for similar data change
3. Master DB system will listen to topics / channel to get change request from other system. It will also act as publisher for other system where it will push same message to different topic / channel after processing it.
4. Consumers (which is on S1-SN) will listen to topics / channels for similar data change which is putted by master DB system. It will consume that data to update their own System DB for Data changes.

In this way you can achieve Data consistency between systems. Only thing you need to take care is delay in this process, which could be arise if any system fails.

Hope I have answer your query.

Answer 3

Maintain one table to keep track of all latest updates (Latest One day updates). Assuming that all updates can be propagated to all other systems via pub/sub system within a couple of minutes eventually.

All systems should contact this primary source before updating the entry (for fields like name, email and contact number)

• If there is already an entry in that primary source table then need to verify that which is the latest one. If their is newer entry then need to alert to the user that there were some recent updates happened to the same record, do you still want to update it.

• If there is no entry then you can easily update the data with the new data and insert one entry into these primary source table.

Whenever any system processes updates via pub-sub then it should update only if there are no new updates happend to the same record.