如何实时更新数据到主仓库

Question

All, Need some info.

We have stores at multiple locations and use client server app installed for sales activity. sales data is stored in database which is setup in all stores... @ end of day - a batch pulls data from all of the store locations and update main warehouse database.

We want to have real time implementation so that whenever there is transcation @ any store... data will update immediately to main warehouse repository.

Any clue as how can we achive real time update of data to main warehouse ?

Thanks in advance...

Answer 1

One approach to this is called replication . There are several ways to do it in SQL Server. You're probably looking for transaction replication or merge replication.

Here's a place to start in the SQL Server 2012 documentation. And here's a fairly recent overview that might be helpful.

Answer 2

You should make sure you understand what "real time" means, and how real time you really need to be. If you are not pre aggregating data and then storing it in the WH, then you should be able to set up replication between the database servers (if they can talk to each other). If you are loading an aggregate, then it gets tricky because you have to merge the measures (facts) into the warehouses existing measures, which is tough. If you don't need true real time, just a slow trickle, then consider simply running your current process on a schedule in sql agent.

Answer 3

First off - why not run the batch multiple times a day. It would not really be "real-time" but might yield good enough real world results.

One option would be to implement master-master replication provided by the SQL engine in use. Though this probably means that some steps need to be taken to guard against duplicate IDs, auto increment mismatch etc. For example we have a master-master system set up so that one produces entries with odd IDs, the other with even.

Another approach could be that all reads are performed against local databases, and all writes are performed into the single remote master. Data would be replicated as a master-slave setup. This would provide best data consistency, but slow network would make any writes slow. We have this kind of a setup implemented atop of the master-master replication as most interaction are reads.

One real world use case I have actually come across for a similar stores/warehouse setup was based on Firebird SQL. Every single table had triggers implemented to store every action on local databases in so called log tables. And there was a replication application running at all times, regularly checking these log tables, updating the data to a remote database and pulling in new data from the remote (which had it's own log tables). But as a downside it was a horror to maintain as triggers needed to be updated when something changed in the database setup and the replication application would fail/hang at times. But data consistency was maintained well and resolved by negative IDs being used for local database and positive for master/remote. But in the end it did not really provide real "real-time".

In the end - there is no one-shoe-fits-all answer and books could probably be written on the topic. Research and Google are your friends.