使用MySQL报告-最简单的查询花费的时间太长

Question

I have a MySQL Table on an Amazon RDS Instance with 250 000 Rows. When I try to

SELECT * FROM  tableName 

without any conditions ( just for testing , the normal query specifies the columns I need, but I need most of them) , the query takes between 20 and 60 seconds to execute. This will be the base query for my report, and the report should run in under 60 seconds, so I think this will not work out (it times out the moment I add the joins). The report runs without any problems in our smaller test environments.

Could it be that the Query is taking so long because MySQL is trying to lock the table and waiting for all writes to finish? There might be quite a lot of writes on this table. I am doing the query on a MySQL slave, since I do not want to lockup the production system with my queries.

• I have no experience with how much rows are much for a relational DB. Are 250 000 Rows with ~30 columns (varchar, date and integer types) much?
• How can I speedup this query (hardware, software, query optimization ...)
• Can I tell MySQL that I do not care that the Data might be inconsistent (It is a snapshot from a Reporting Database)
• Is there a chance that this query will run under 60 seconds, or do I have to adjust my goals?

Answer 1

A table with 250,000 rows is not too big for MySQL at all.

However, waiting for those rows to be returned to the application does take time. That is network time, and there are probably a lot of hops between you and Amazon.

Unless your report is really going to process all the data, check the performance of the database with a simpler query, such as:

select count(*) from table;

EDIT:

Your problem is unlikely to be due to the database. It is probably due to network traffic. As mentioned in another answer, streaming might solve the problem. You might also be able to play with the data formats to get the total size down to something more reasonable.

A last-resort step would be to save the data in a text file, compress the file, move it over, and uncompress it. Although this sounds like a lot of work, you might get 5x - 10x compression on the data, saving oodles of time on the transmission and still have a large improvement in performance with the rest of the processing.

Answer 2

Remember that MySQL has to prepare your result set and transport it to your client. In your case, this could be 200MB of data it has to shuttle across the connection, so 20 seconds is not bad at all. Most libraries, by default, wait for the entire result being received before forwarding it to the application.

To speed it up, fetch only the columns you need, or do it in chunks with LIMIT . SELECT * is usually a sign that someone's being super lazy and not optimizing at all.

If your library supports streaming resultsets, use that, as then you can start getting data almost immediately. It'll allow you to iterate on rows as they come in without buffering the entire result.

Answer 3

I got updated specs from my client and was able to reduce the amount of users returned to 250, which goes (with a lot of JOINS) though in 60 seconds.

So maybe the answer is really: Try to not dump a whole table with a query, fetch only the exact data your need. The Client has SQL access, and he will have to update his queries, so only relevant users are returned.

Answer 4

I should never really use * as a wildcard. Choose the fields that you actually want and then create an index of these fields combined.

Answer 5

If you have thousands of rows, another option is implement pagination. If result data directly using for report , no one can look more than 100 rows in single shot.