通过优化对MySQL分组-避免使用tmp表和/或文件排序

Question

I have a slow query, without the group by is fast (0.1-0.3 seconds), but with the (required) group by the duration is around 10-15s.

The query joins two tables, events (near 50 million rows) and events_locations (5 million rows).

Query:

SELECT  `e`.`id` AS `event_id`,`e`.`time_stamp` AS `time_stamp`,`el`.`latitude` AS `latitude`,`el`.`longitude` AS `longitude`,
        `el`.`time_span` AS `extra`,`e`.`entity_id` AS `asset_name`, `el`.`other_id` AS `geozone_id`,
        `el`.`group_alias` AS `group_alias`,`e`.`event_type_id` AS `event_type_id`,
        `e`.`entity_type_id`AS `entity_type_id`, el.some_id
FROM events e
INNER JOIN events_locations el ON el.event_id = e.id
WHERE 1=1       
    AND el.other_id = '1'  
    AND time_stamp >= '2018-01-01'  
    AND time_stamp <= '2019-06-02'
GROUP BY `e`.`event_type_id` , `el`.`some_id` , `el`.`group_alias`;

Table events:

CREATE TABLE `events` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `event_type_id` int(11) NOT NULL,
  `entity_type_id` int(11) NOT NULL,
  `entity_id` varchar(64) NOT NULL,
  `alias` varchar(64) NOT NULL,
  `time_stamp` datetime NOT NULL,
  PRIMARY KEY (`id`),
  KEY `entity_id` (`entity_id`),
  KEY `event_type_idx` (`event_type_id`),
  KEY `idx_events_time_stamp` (`time_stamp`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Table events_locations

CREATE TABLE `events_locations` (
  `event_id` bigint(20) NOT NULL,
  `latitude` double NOT NULL,
  `longitude` double NOT NULL,
  `some_id` bigint(20) DEFAULT NULL,
  `other_id` bigint(20) DEFAULT NULL,
  `time_span` bigint(20) DEFAULT NULL,
  `group_alias` varchar(64) NOT NULL,
  KEY `some_id_idx` (`some_id`),
  KEY `idx_events_group_alias` (`group_alias`),
  KEY `idx_event_id` (`event_id`),
  CONSTRAINT `fk_event_id` FOREIGN KEY (`event_id`) REFERENCES `events` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

The explain:

+----+-------------+-------+--------+---------------------------------+---------+---------+-------------------------------------------+----------+------------------------------------------------+
| id | select_type | table | type   | possible_keys                   | key     | key_len | ref                                       | rows     | Extra                                          |
+----+-------------+-------+--------+---------------------------------+---------+---------+-------------------------------------------+----------+------------------------------------------------+
| 1  | SIMPLE      | ea    | ALL    | 'idx_event_id'                  | NULL    | NULL    | NULL                                      | 5152834  | 'Using where; Using temporary; Using filesort' |
| 1  | SIMPLE      | e     | eq_ref | 'PRIMARY,idx_events_time_stamp' | PRIMARY | '8'     | 'name.ea.event_id'                        | 1        |                                                |
+----+-------------+----------------+---------------------------------+---------+---------+-------------------------------------------+----------+------------------------------------------------+
2 rows in set (0.08 sec)

From the doc :

Temporary tables can be created under conditions such as these:

If there is an ORDER BY clause and a different GROUP BY clause, or if the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue, a temporary table is created.

DISTINCT combined with ORDER BY may require a temporary table.

If you use the SQL_SMALL_RESULT option, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.

I already tried:

• Create an index by ' el . some_id , el . group_alias '
• Decrease the varchar size to 20
• Increase the size of sort_buffer_size and read_rnd_buffer_size;

Any suggestions for performance tuning would be much appreciated!

Answer 1

In your case events table has time_span as indexing property. So before joining both tables first select required records from events table for specific date range with required details. Then join the event_location by using table relation properties.

Check your MySql Explain keyword to check how does your approach your table records. It will tell you how much rows are scanned for before selecting required records.

Number of rows that are scanned also involve in query execution time. Use my below logic to reduce the number of rows that are scanned.

SELECT  
    `e`.`id` AS `event_id`,
    `e`.`time_stamp` AS `time_stamp`,
    `el`.`latitude` AS `latitude`,
    `el`.`longitude` AS `longitude`,
    `el`.`time_span` AS `extra`,
    `e`.`entity_id` AS `asset_name`, 
    `el`.`other_id` AS `geozone_id`,
    `el`.`group_alias` AS `group_alias`,
    `e`.`event_type_id` AS `event_type_id`,
    `e`.`entity_type_id` AS `entity_type_id`, 
    `el`.`some_id` as `some_id`
FROM 
    (select
        `id` AS `event_id`,
        `time_stamp` AS `time_stamp`,
        `entity_id` AS `asset_name`,
        `event_type_id` AS `event_type_id`,
        `entity_type_id` AS `entity_type_id`
    from
        `events` 
    WHERE
        time_stamp >= '2018-01-01'  
        AND time_stamp <= '2019-06-02'
    ) AS `e`    
    JOIN `events_locations` `el` ON `e`.`event_id` = `el`.`event_id`
WHERE     
    `el`.`other_id` = '1'      
GROUP BY 
    `e`.`event_type_id` , 
    `el`.`some_id` , 
    `el`.`group_alias`;

Answer 2

The relationship between these tables is 1:1, so, I asked me why is a group by required and I found some duplicated rows, 200 in 50000 rows. So, somehow, my system is inserting duplicates and someone put that group by (years ago) instead of seek of the bug.

So, I will mark this as solved, more or less...