[英]MySQL query optimisation, big table, using temporary filesort
我需要您的帮助来优化查询。 一个表是一个具有数百万个条目的日志表,我尝试将查询破坏为<1s。 我的查询应该给出总体概述,因此应该很快。 我相信我可以借助脚本在列表上进行多个简单查询。 但是脚本可以做什么,mysql我可以吗?我至少希望如此。 也许并不是所有部分都得到最好的利用,但是我陷入了一个查询,该查询使一个临时表和文件排序(我发现这确实很糟糕)。 在阅读时,我发现使用了一些整洁且位置良好的索引,但是现在我陷入了一个特定的问题。
让我向您展示我的最终查询及其结果:
SELECT
ps.SERVER_ID,
ps.FULLNAME,
SUM(CASE WHEN pml.ID_TYPE = 3 THEN 1 ELSE 0 END) 'amount_warning',
SUM(CASE WHEN pml.ID_TYPE = 4 THEN 1 ELSE 0 END) 'amount_error',
SUM(CASE WHEN pml.ID_TYPE = 5 THEN 1 ELSE 0 END) 'amount_alert',
SUM(CASE WHEN pml.ID_TYPE = 7 THEN 1 ELSE 0 END) 'amount_critical'
FROM
PAR_SERVER ps
INNER JOIN
PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID
WHERE
pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY )
GROUP BY
ps.SERVER_ID;
这是我得到的:
mysql> [thequeryabove]
[...]
59 rows in set (11.69 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | ALL | SERVER_ID,SERVER_ID-ID_TYPE | NULL | NULL | NULL | 4014447 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
这是我当前的表格设置:
mysql> describe PAR_SERVER;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| SERVER_ID | int(255) | NO | PRI | NULL | auto_increment |
| FULLNAME | varchar(255) | YES | | NULL | |
| SHORTNAME | varchar(255) | YES | MUL | NULL | |
+----------------+--------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
mysql> show indexes from PAR_SERVER;
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_SERVER | 0 | PRIMARY | 1 | SERVER_ID | A | 142 | NULL | NULL | | BTREE | |
| PAR_SERVER | 1 | shortname | 1 | SHORTNAME | A | 142 | NULL | NULL | YES | BTREE | |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
mysql> select count(*) from PAR_SERVER;
+----------+
| count(*) |
+----------+
| 142 |
+----------+
1 row in set (0.00 sec)
mysql> describe PAR_MONITORINGv2_LOG;
+--------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+----------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| ID_TYPE | int(11) | NO | MUL | NULL | |
| ID_SERVICE | int(11) | NO | MUL | NULL | |
| SERVER_ID | int(11) | NO | MUL | NULL | |
| MESSAGE | tinytext | NO | | NULL | |
| CREATED_DATE | datetime | NO | | NULL | |
+--------------+----------+------+-----+---------+----------------+
6 rows in set (0.00 sec)
mysql> show indexes from PAR_MONITORINGv2_LOG;
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_MONITORINGv2_LOG | 0 | PRIMARY | 1 | ID | A | 3998188 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | ID_TYPE | 1 | ID_TYPE | A | 7 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | ID_SERVICE | 1 | ID_SERVICE | A | 5 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID | 1 | SERVER_ID | A | 66 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID-ID_TYPE | 1 | SERVER_ID | A | 66 | NULL | NULL | | BTREE | |
| PAR_MONITORINGv2_LOG | 1 | SERVER_ID-ID_TYPE | 2 | ID_TYPE | A | 258 | NULL | NULL | | BTREE | |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
6 rows in set (0.00 sec)
mysql> select count(*) from PAR_MONITORINGv2_LOG;
+----------+
| count(*) |
+----------+
| 3998386 |
+----------+
1 row in set (0.00 sec)
这是逐步打破我的查询的时间结果。 修复每个零件花费了这么长时间后,我可能会逐步进行。 但是目前,对于该问题,当前仅运行时间为2.30秒的查询才有意义。
mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID WHERE pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY ) GROUP BY ps.SERVER_ID;
[...]
59 rows in set (6.41 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | ALL | SERVER_ID,SERVER_ID-ID_TYPE | NULL | NULL | NULL | 4014788 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID GROUP BY ps.SERVER_ID;
[...]
59 rows in set (2.30 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | pml | index | SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID | 4 | NULL | 4015694 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | ps | eq_ref | PRIMARY | PRIMARY | 4 | database.pml.SERVER_ID | 1 | |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)
mysql> SELECT pml.SERVER_ID FROM PAR_MONITORINGv2_LOG pml GROUP BY pml.SERVER_ID;
[...]
65 rows in set (0.00 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
| 1 | SIMPLE | pml | range | NULL | SERVER_ID | 4 | NULL | 67 | Using index for group-by |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
1 row in set (0.00 sec)
我可以通过定义(SERVER_ID,ID_TYPE)的索引来大大改善查询,如下面的示例查询所确认:
mysql> SELECT count(*) 'count_warnings' FROM PAR_MONITORINGv2_LOG pml WHERE pml.SERVER_ID = 191 AND pml.ID_TYPE = 3 GROUP BY pml.SERVER_ID;
[...]
1 row in set (0.01 sec)
mysql> explain [thequeryabove]
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | pml | ref | ID_TYPE,SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID-ID_TYPE | 8 | const,const | 10254 | Using index |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
1 row in set (0.00 sec)
我现在陷入了最细分的查询中,执行时间仅为2.30秒。 我不知道如何在没有任何where子句的查询中使用索引。
通过在PAR_MONITORINGv2_LOG(CREATED_DATE,SERVER_ID,ID_TYPE)上添加复合索引,您的查询无疑将获得最大的收益。 但是,我建议即使在CREATED_DATE
简单的索引也可以CREATED_DATE
提高性能。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.