从具有多个线程的单个数据库表中获取数据的最佳方法？

Question

we have a system where we collect data every second on user activity on multiple web sites. we dump that data into a database X (say MS SQL Server). we now need to fetch data from this single table from daatbase X and insert into database Y (say mySql).

we want to fetch time based data from database X through multiple threads so that we fetch as fast as we can. Once fetched and stored in database Y, we will delete data from database X.

Are there any best practices on this sort of design? any specific things to take care on table design like sharing or something? Are there any other things that we need to take care to make sure we fetch it as fast as we can from threads running on multiple machines?

Thanks in advance! Ravi

Answer 1

If you are moving data from one database to another, you will not gain any advantages by having multiple threads doing the work. It will only increase contention.

If both databases are of the same type, you should be looking into the vendors specific tools for replication. This will basically always outperform homegrown solutions.

If the databases are different (vendors), you have to decide upon an efficient mechanism for

1. identifying new/updated/deleted rows (Triggers, range based queries, full dumps)
2. transporting the data (unload to file & FTP, pull/push from a program)
3. loading the data on the other database (import, bulk insert)

Without more details, it's impossible to be more specific than that. Oh, and the two most important considerations that will influence your choice are:

1. What is the expected data volume?
2. Longest acceptable delay between row creation in source DB and availability in Target DB

Answer 2

I would test (by measurement) your assumption that multiple slurper threads will speed things up. Without being more specific in your question, it looks like you want to do an ETL (extract transform load) process with your database, these are pretty efficient when you let the database specific technology handle it, especially if you're interested in aggregation etc.

Answer 3

There are two levels of concern of your issue:

1. The transaction between these two database:

   This is important because you would delete database from source database. You must ensure that only remove data from X while the database has been stored into Y successfully. On the other side, your must ensure that the deletion of data from X must be successful to prevent re-insert same data into Y.

2. The performance of transferring data:

   If the X database has incoming data whenever, which is a online database, it is not a good practice that just collect data, store to Y, and delete them. Planning a size of batch, the program starts a transaction for that batch; running the program repeatedly until the number of data in X is under the size of batch.

In both of databases, your should add a table to record the batch for processing. There are three states in processing.

INIT - The start of batch, this value should be synchronized between two databases
COPIED - In database Y, the insertion of data and the update of this status should be in one transaction.
FINISH - In database X, the deletion of data and the update of this status should be in on transaction.

When the programing is running, it first checks the batches in 'INIT' or 'COPIED' state and restarts the session to process.

• If X has an "INIT" record and Y don't, just insert the same INIT record to Y, then perform the insertion to Y.
• If a record in Y is "COPIED" and X is "INIT", just update the state of X to "COPIED", then perform the deletion to X.
• If a record in X is "FINISH" and the corresponding record in Y is "COPIED", just update the the state of Y to "FINISH".

In conclusion, processing data at a batch would give you a chance to optimize such transferring between two databases. The number of batch size dominates the efficiency of transforming and depends on two factors: how those databases concurrently used by other operation and the tuning parameter of your databases. In general situation, the write-throughput of Y is likely the bottleneck of processing.

Answer 4

Threads are not the way to go. The database(s) is the bottleneck here. Multiple threads will only increase contention. Even if 10 processes are jamming data into SQL Server, a single thread (rather than many) can pull it out faster. There is absolutely no doubt about that.

The SELECT itself can cause locks in the main table, reducing the throughput of the INSERTs, so I would "get in and get out" as fast as possible. If it were me, I would:

1. SELECT the rows based on a range query (date, recno, whatever), dump them into a file, and close the result set (cursor).
2. DELETE the rows based on the same range query.
3. Then process the dump. If possible, the dump format should be amenable to bulk-load into MySQL.

I don't want to beat up your architecture, but overall the design sounds problematic. SELECTing and DELETEing rows from a table undergoing a high INSERTion rate is going to create huge locking issues. I would be looking at "double-buffering" the data in the SQL Server.

For example, every minute the inserts switch between two tables. For example, in the first minute INSERTs go into TABLE_1, but when the minute rolls over they start INSERTing into TABLE_2, the next minute back to TABLE_1, and so forth. While INSERTS are going into TABLE_2, SELECT everything from TABLE_1 and dump it into MySQL (as efficiently as possible), then TRUNCATE the table (deleting all rows with zero penalty). This way, there is never lock-contention between the readers and writers.

Coordinating the rollover point of between TABLE_1 and TABLE_2 is the tricky part. But it can be done automatically through a clever use of SQL Server Partitioned Views.