我需要优化以下SQL查询

Question

i need to optimize the following sql query as i have been waiting for the past 45 minutes for it to execute.

    create database comparelogs;
    use comparelogs;
    create table compare(field varchar(1080));

    load data local infile 'c:/path/text.txt'
    into table compare
    lines terminated by '\n';

the following query is the one that is consuming time.

    SELECT
    field,
    (SELECT COUNT(*) FROM compare T2 WHERE T2.field = T1.field) AS Count,
    @row_num := if(@prev_value=field,@row_num+1,1) as Occurrence,
    @prev_value := field as previous_value
    FROM compare T1
    order by field;

    input data:(from text.txt file)
    /Jols/AAP/AAP.36/Ads\AdS_7.75x10_1.29.75_k
    /Jols/AAP/AAP.36/Ads\PhyJobPl4x1034_BW_1.5.pdf
    /Jls/AAP/AAP.36Ads\Cusins BW_rint_FPbw_1.3.pdf
    /Jouals/AAP/AAP.36Ads\GeneryjWebdBW_1.6.pdf AAR356BF
    /Jals/AAP/AAP.36Ads\GeneralWyjyjyAW_1.6.pdf AAR356BF
    /Jls/AAP/AAP.36Ads\XtraCredit filler ad - PROPRIETARY ONLY_4C_1.0.pdf
    /Jos/AAP/AAP.36Ads\AAP_May-Jun_10_Havel's_EchoBlock_1.0.pdf
    /Jls/AAP/AAP.36//As\Roc76SolidAdbw_1.0.pdf
    /Jls/AAP/AAP.36//A\RAPM_fi7ul ad_BW_1.0.pdf
    /Jls/AAP/AAP.36/As\AAP_1_11_Bjhkytaun_Dyjidance_1.0.pdf
    /Jls/AAP/AAP.36/s\AAPyjs36_yj.2.pdf
     ..........................................upto 200000 records.

Its been more than 45 minutes and it is still executing how can i optimize this to get the result fast. I'm using mysql56.

Answer 1

This is a good amout of data you are loading. Which will take some time. The same questions below might help you

http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/

another link regarding the performance

http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/

Answer 2

I Think MySQL is executing SELECT statement. Why don't you run SHOW PROCESSLIST (I think Loading 200k rows using LOAD DATA INFILE would not take long.)

In my experience, correlated sub-query is very slow in MySQL. Could you try this?

You can test here. http://www.sqlfiddle.com/#!2/ebcfe/7

SET @row_num := 0;
SET @prev_value := '';

SELECT field,
    @row_num := if(@prev_value=field,@row_num+1,1) as Occurrence,
    @prev_value := field as previous_value
FROM
(
    SELECT
      T1.field, cnt
    FROM compare T1 INNER JOIN
        (
            SELECT field, COUNT(*) AS cnt
            FROM compare
            GROUP BY field
        ) AS T2 ON T1.field = T2.field
    ORDER BY T1.field
) x;
+--------+------------+----------------+
| field  | Occurrence | previous_value |
+--------+------------+----------------+
| field1 |          1 | field1         |
| field2 |          1 | field2         |
| field2 |          2 | field2         |
| field3 |          1 | field3         |
| field3 |          2 | field3         |
| field3 |          3 | field3         |
| field4 |          1 | field4         |
| field4 |          2 | field4         |
| field4 |          3 | field4         |
| field4 |          4 | field4         |
+--------+------------+----------------+