使用MySQL中的100万条记录创建报表并显示在Java JSP页面中

Question

I am working on a MySQL database with 3 tables - workout_data, excercises and sets tables. I'm facing issues related to generating reports based on these three tables.

To add more information, a number of sets make up an excercise and a number of excercises will be a workout. I currently have the metrics to which a report is to be generated from the data in these tables. I've to generate reports for the past 42 days including this week. The queries run for a long time by the time I get the report by joining these tables.

For example - the sets table has more than 1 million records just for the past 42 days. The id in this table is the excercise_id in excercise table. The id of excercise table is the workout_id in workout_data table.

I'm running this query and it takes more than 10 minutes to get the data. I have to prepare a report and show it to the user in the browser. But due to this long running query the webpage times out and the user is not able to see the report.

Any advice on how to achieve this?

        SELECT REPORTSETS.USER_ID,REPORTSETS.WORKOUT_LOG_ID,
               REPORTSETS.SET_DATE,REPORTSETS.EXCERCISE_ID,REPORTSETS.SET_NUMBER 
          FROM EXCERCISES 
    INNER JOIN REPORTSETS ON EXCERCISES.ID=REPORTSETS.EXCERCISE_ID 
         where user_id=(select id from users where email='testuser1@gmail.com') 
           and substr(set_date,1,10)='2013-10-29' 
      GROUP BY REPORTSETS.USER_ID,REPORTSETS.WORKOUT_LOG_ID,
               REPORTSETS.SET_DATE,REPORTSETS.EXCERCISE_ID,REPORTSETS.SET_NUMBER

Answer 1

Comments on your SQL that you might want to look into:

1) Do you have an index on USER_ID and SET_DATE?

2) Your datatype for SET_DATE looks wrong, is it a varchar? Storing it as a date will mean that the db can optimise your search much more efficiently. At the moment the substring method will be called countless times per query as it has to be run for every row returned by the first part of your where clause.

3) Is the group by really required? Unless I'm missing something the 'group by' part of the statement brings nothing to the table ;)

Answer 2

Two things:

First, You have the following WHERE clause item to pull out a single day's data.

  AND substr(set_date,1,10)='2013-10-29'

This definitively defeats the use of an index on the date. If your set_date column has a DATETIME datatype, what you want is

  AND set_date >= `2013-10-09`
  AND set date <  `2013-10-09` + INTERVAL 1 DAY

This will allow the use of a range scan on an index on set_date. It looks to me like you might want a compound index on (user_id, set_date) . But you should muck around with EXPLAIN to figure out whether that's right.

Second, you're misusing GROUP BY . That clause is pointless unless you have some kind of summary function like SUM() or GROUP_CONCAT() in your query. Do you want ORDER BY ?

Answer 3

It should make a significant difference if you could store the date either as a date, or in the format you need to make the comparison. Performing a substr() call on every date must be time consuming.

Answer 4

Surely the suggestions with tuning the query would help to improve the query speed. But I think the main point here is what can be done with more than 1 million plus records before session timed out . What if you have like 2 or 3 million records, will some performance tuning solve the problem? I don't think so. So:

1) If you want to display on browser, use pagination and query (for example) the first 100 record.
2) If you want to generate a report (like pdf), then use asynchronous method (JMS)