MySQL：如何从MySQL中提取大量数据而不会使其窒息？

Question

My colleague run a script that pulls data from the db periodically. He is using the query:

SELECT url, data FROM table LIMIT {} OFFSET {}'.format( OFFSET, PAGE * OFFSET

We use Amazon AURORAS and he has his own slaves server but everytime it touches 98%+

Table have millions of records.

Would it be nice if we go for sqldump instead of SQL queries for fetching data?

The options come in my mind are:

• SQL DUMP of selective tables( not sure of benchmark)
• Federate tables based on certain reference(date, ID etc)

Thanks

Answer 1

I'm making some fairly big assumptions here, but from

without choking it

I'm guessing you mean that when your colleague runs the SELECT to grab the large amount of data, the database performance drops for all other operations - presumably your primary application - while the data is being prepared for export.

You mentioned SQL Dump so I'm also assuming that this colleague will be satisfied with data that is roughly correct, ie: it doesn't have to be up to the instant transactionally correct data. Just good enough for something like analytics work.

If those assumptions are close, your colleague and your database might benefit from

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

This line of code should be used carefully and almost never in a line of business application but it can help people querying the live database with big queries, as long as you fully understand the implications.

To use it, simply start a transaction and put this line before any queries you run.

The 'choking'

What you are seeing when your colleague runs a large query is record locking. Your database engine is - quite correctly - set up to provide an accurate view of your data, at any point. So, when a large query comes along the database engine first waits for all write locks (transactions) to clear, runs the large query and holds all future write locks until the query has run.

This actually happens for all transactions, but you only really notice it for the big ones.

What READ UNCOMMITTED does

By setting the transaction isolation level to READ UNCOMMITTED, you are telling the database engine that this transaction doesn't care about write locks and to go ahead and read anyway.

This is known as a 'dirty read', in that the long-running query could well read a table with a write lock on it and will ignore the lock. The data actually read could be the data before the write transaction has completed, or a different transaction could start and modify records before this query gets to it.

The data returned from anything with READ UNCOMMITTED is not guaranteed to be correct in the ACID sense of a database engine, but for some use cases it is good enough .

What the effect is

Your large queries magically run faster and don't lock the database while they are running.

Use with caution and understand what it does before you use it though.

MySQL Manual on transaction isolation levels