使用 GROUP_CONCAT 的 MySql 查询优化

Question

This query has been reported in slow logs which is clearly taking query_time: 00:00:20 seconds to execute. It is expected to send one row. The number of rows examined by this query is 9702034. Explaining query does not give much clue on how this query can be optimized. :

      SELECT DISTINCT `ecu`.*, `ecuad`.`google_address`, `ecuad`.`town`, `ecuad`.`region`, `ecuad`.`zip`, `ecuad`.`country_id`, `ecuad`.`lat`, `ecuad`.`long`, `bc`.`name` AS `company_name`, GROUP_CONCAT( CONCAT(euad.assign_to_company_id,"_",euad.assigned_to)) AS `assign_company_dm_id`, GROUP_CONCAT( distinct edhp.location_id ) AS `location_id`, GROUP_CONCAT( distinct edhp.job_type ) AS `job_type`, GROUP_CONCAT( distinct edhp2.industry_id ) AS `industry_id`, GROUP_CONCAT( distinct edhp2.id ) AS `hiringpref_id)`, GROUP_CONCAT( distinct edhpskl.skill_id ) AS `skill_id`, GROUP_CONCAT( distinct edhpskl.category_id ) AS `category_id`, `ecli`.`login_date` FROM `es_company_users` AS `ecu`
 LEFT JOIN `es_company_user_addresses` AS `ecuad` ON ecuad.dm_id = ecu.company_user_id and ecuad.base_company_id = ecu.base_company_id
 LEFT JOIN `base_companies` AS `bc` ON bc.id = ecu.base_company_id
 LEFT JOIN `es_user_assigned_dm` AS `euad` ON euad.foreign_id = ecu.company_user_id and euad.base_company_id = ecu.base_company_id
 LEFT JOIN `es_dm_hiring_preference` AS `edhp` ON edhp.dm_id = ecu.company_user_id and edhp.base_company_id = ecu.base_company_id AND edhp.location_id > 0
 LEFT JOIN `es_dm_hiring_preference` AS `edhp2` ON edhp2.dm_id = ecu.company_user_id and edhp2.base_company_id = ecu.base_company_id
 LEFT JOIN `es_dm_hiring_skills` AS `edhpskl` ON edhpskl.hiring_preference_id = edhp2.id AND edhpskl.skill_id > 0
 LEFT JOIN `es_edclient_login_info` AS `ecli` ON ecu.company_user_id = ecli.company_user_id AND ecli.base_company_id = ecu.base_company_id AND ecli.`login_date` = (SELECT MAX(t2.login_date) FROM es_edclient_login_info t2 WHERE (t2.company_user_id = 780)) WHERE (ecu.id = '636') AND (ecu.company_user_id = '780')

Can you help me out to improve this query performance please.

Here is the Explain of this query

+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
| id | select_type | table   | type  | possible_keys                              | key             | key_len | ref                  | rows | Extra       |
+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
|  1 | PRIMARY     | ecu     | const | PRIMARY,company_user_id                    | PRIMARY         | 4       | const                |    1 | NULL        |
|  1 | PRIMARY     | ecuad   | ref   | dm_id                                      | dm_id           | 10      | const,const          |  667 | NULL        |
|  1 | PRIMARY     | bc      | const | PRIMARY                                    | PRIMARY         | 4       | const                |    1 | NULL        |
|  1 | PRIMARY     | euad    | ref   | base_company_id                            | base_company_id | 10      | const,const          |    2 | NULL        |
|  1 | PRIMARY     | edhp    | ref   | base_company_id                            | base_company_id | 10      | const,const          |   60 | Using where |
|  1 | PRIMARY     | edhp2   | ref   | base_company_id                            | base_company_id | 10      | const,const          |   60 | NULL        |
|  1 | PRIMARY     | edhpskl | ref   | base_company_id                            | base_company_id | 5       | singledb_ed.edhp2.id |    1 | Using where |
|  1 | PRIMARY     | ecli    | ref   | base_company_id,company_user_id,login_date | login_date      | 5       | const                |    1 | Using where |
|  2 | SUBQUERY    | t2      | ref   | company_user_id                            | company_user_id | 4       | const                |    2 | NULL        |
+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
9 rows in set (0.00 sec)

Here is the create info of this query

CREATE TABLE `es_dm_hiring_skills` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `hiring_preference_id` int(11) DEFAULT NULL,
  `category_id` int(11) DEFAULT NULL,
  `skill_id` int(11) DEFAULT NULL,
  `skill_matching_type` int(5) DEFAULT '1',
  `created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `rs_migrated_id` int(11) DEFAULT NULL,
  `is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
  `eb_reff_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `base_company_id` (`hiring_preference_id`)
) ENGINE=InnoDB AUTO_INCREMENT=460021 DEFAULT CHARSET=utf8;


CREATE TABLE `es_dm_hiring_preference` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `base_company_id` int(11) DEFAULT NULL,
  `dm_id` int(11) DEFAULT NULL,
  `category_id` int(11) DEFAULT NULL,
  `job_role_id` int(11) DEFAULT NULL,
  `skill_id` int(11) DEFAULT NULL,
  `location_id` int(11) DEFAULT NULL,
  `office_id` int(11) DEFAULT NULL COMMENT 'id of company_timezone_setting table',
  `job_type` tinyint(1) DEFAULT NULL,
  `industry_id` varchar(255) DEFAULT NULL,
  `created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `is_updated` tinyint(1) DEFAULT '0' COMMENT '0=not updated, 1=updated, 2=updated by cron',
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
  `merge_account_id` int(11) DEFAULT NULL,
  `rs_migrated_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `base_company_id` (`base_company_id`,`dm_id`)
) ENGINE=InnoDB AUTO_INCREMENT=415866 DEFAULT CHARSET=utf8;

CREATE TABLE `es_user_assigned_dm` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `base_company_id` int(11) DEFAULT NULL,
  `assign_to_company_id` int(11) NOT NULL,
  `foreign_id` int(11) DEFAULT NULL,
  `assigned_to` int(11) DEFAULT NULL COMMENT 'base user id, candidate is assigned to which user? ',
  `assigned_type` int(2) DEFAULT '2' COMMENT '1= My Network, 2= Auto assignee',
  `to_type` tinyint(1) DEFAULT '1' COMMENT '1 = Ed manager, 2= Marketing Consultant',
  `assigned_by` int(11) DEFAULT NULL COMMENT 'dm assigned by which user',
  `tiering_level` int(11) DEFAULT NULL,
  `organisation_id` int(11) DEFAULT NULL,
  `assigned_date` datetime DEFAULT NULL,
  `modified_by` int(11) DEFAULT NULL,
  `modified_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `branch_id` int(11) DEFAULT NULL,
  `merge_account_id` int(11) DEFAULT NULL,
  `rs_migrated_id` int(11) DEFAULT NULL,
  `is_eb_migrated` tinyint(4) DEFAULT '0',
  `eb_reff_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `base_company_id` (`base_company_id`,`foreign_id`)
) ENGINE=InnoDB AUTO_INCREMENT=493547 DEFAULT CHARSET=utf8;

CREATE TABLE `base_companies` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `industry_id` int(11) DEFAULT NULL,
  `country_id` int(11) DEFAULT NULL,
  `supercrm_account_id` int(11) DEFAULT NULL,
  `name` varchar(250) NOT NULL,
  `status` tinyint(1) NOT NULL,
  `size` tinyint(1) NOT NULL,
  `timezone` varchar(255) DEFAULT NULL,
  `logo` varchar(255) DEFAULT NULL,
  `theme` varchar(255) NOT NULL,
  `namespace` varchar(255) DEFAULT NULL,
  `db_user` varchar(255) DEFAULT NULL,
  `db_pass` varchar(255) DEFAULT NULL,
  `db_host` varchar(255) NOT NULL,
  `db_port` int(4) NOT NULL,
  `db_socket` varchar(255) NOT NULL,
  `cdn_url` varchar(255) DEFAULT NULL,
  `cdn_ssl_url` varchar(256) NOT NULL,
  `client_portal_flag` tinyint(1) DEFAULT NULL,
  `client_company_id` int(11) DEFAULT NULL COMMENT 'Parent company id from which this have activated as client portal',
  `mod_date` timestamp NULL DEFAULT NULL,
  `created_date` datetime DEFAULT NULL,
  `show_ed_manager` tinyint(1) DEFAULT '0' COMMENT 'temp column to flag Companys for Ed Manager',
  `is_migrate_rpo` int(4) NOT NULL DEFAULT '0',
  `is_sourcechain_account` smallint(2) DEFAULT '0' COMMENT '0=> Normal Account, 1=> Sourcechain account',
  `sourcechain_from_name` varchar(255) DEFAULT NULL,
  `sourcechain_email` varchar(255) DEFAULT NULL,
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `organisation_portal_flag` tinyint(4) DEFAULT '0' COMMENT '1 - CB, 2 - EB, 3 - TR ',
  `blocked_organisation_flag` int(4) DEFAULT NULL COMMENT '1 - Blocked for new organisation, 2 - Removed Blocking for new organisation',
  `rs_migrated_id` bigint(20) DEFAULT NULL,
  `is_eb_migrated` tinyint(4) DEFAULT '0',
  `eb_reff_id` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `status` (`status`),
  KEY `base_companies` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=122873 DEFAULT CHARSET=utf8;

CREATE TABLE `es_company_user_addresses` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `dm_id` int(11) DEFAULT NULL,
  `base_company_id` int(11) DEFAULT NULL,
  `office_id` int(11) DEFAULT NULL COMMENT 'Primary key(id) of company_timezone_setting',
  `google_address` varchar(255) DEFAULT NULL,
  `type` int(11) DEFAULT '2',
  `country_id` int(11) DEFAULT NULL,
  `street1` tinytext,
  `street2` tinytext,
  `street3` tinytext,
  `town` varchar(150) DEFAULT NULL,
  `region` varchar(150) DEFAULT NULL,
  `closest_city` varchar(150) DEFAULT NULL,
  `zip` varchar(20) DEFAULT NULL,
  `lat` float(10,6) DEFAULT NULL,
  `long` float(10,6) DEFAULT NULL,
  `last_mod_user_id` int(11) NOT NULL,
  `mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `created_user_id` int(11) NOT NULL,
  `created_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
  `eb_reff_id` int(11) DEFAULT NULL,
  `updated_geo` tinyint(2) DEFAULT '0',
  `merge_account_id` int(11) DEFAULT NULL,
  `rs_migrated_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `dm_id` (`dm_id`,`base_company_id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=434884 DEFAULT CHARSET=utf8;

CREATE TABLE `es_edclient_login_info` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `base_company_id` int(11) NOT NULL,
  `company_user_id` int(11) NOT NULL,
  `company_name` varchar(255) NOT NULL,
  `user_email` varchar(255) NOT NULL,
  `remote_ip` varchar(255) NOT NULL,
  `assign_to_id` int(11) DEFAULT NULL,
  `aasigned_to_company_id` int(11) NOT NULL,
  `assign_to_email` int(11) DEFAULT NULL,
  `login_date` datetime NOT NULL,
  `merge_account_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `base_company_id` (`base_company_id`),
  KEY `company_user_id` (`company_user_id`),
  KEY `login_date` (`login_date`)
) ENGINE=InnoDB AUTO_INCREMENT=101621 DEFAULT CHARSET=utf8;

CREATE TABLE `es_company_users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `base_company_id` int(11) DEFAULT NULL,
  `base_user_id` int(11) DEFAULT NULL,
  `company_user_id` int(11) DEFAULT NULL,
  `es_client_candidate_link_id` int(11) DEFAULT NULL,
  `branch_id` int(11) DEFAULT NULL,
  `client_portal_id` int(11) DEFAULT NULL COMMENT 'client portal id if user is of client portal',
  `crm_people_id` int(11) DEFAULT NULL COMMENT 'client user id-people_id',
  `education_level` int(11) DEFAULT NULL,
  `division_id` int(11) DEFAULT NULL,
  `job_level` int(11) DEFAULT NULL,
  `fname` varchar(50) DEFAULT NULL,
  `lname` varchar(50) DEFAULT NULL,
  `title` tinyint(4) DEFAULT NULL,
  `job_title` varchar(100) DEFAULT NULL,
  `sex` tinyint(1) DEFAULT NULL,
  `dob` datetime DEFAULT NULL,
  `country_id` int(4) DEFAULT NULL COMMENT 'no constraint should be made on this',
  `picture` varchar(255) DEFAULT NULL,
  `timezone` varchar(255) DEFAULT 'Europe/Berlin',
  `function_id` int(11) DEFAULT NULL,
  `default_module_id` int(11) DEFAULT NULL,
  `status` tinyint(1) DEFAULT NULL COMMENT 'verified, activated, deactivated, approved',
  `endorsed_user` int(1) DEFAULT '1',
  `endorsed_dm_status` int(11) DEFAULT '0',
  `is_private_dm` tinyint(1) DEFAULT '0' COMMENT '0 : Not private DM, 1 : Private DM ',
  `marketplace_status` tinyint(1) NOT NULL DEFAULT '1' COMMENT '0-Deactive, 1-Active',
  `flag_dm_feedback` tinyint(1) DEFAULT '0',
  `rs_company_id` int(11) DEFAULT NULL,
  `source_id` int(11) DEFAULT NULL COMMENT 'Base Company id',
  `source_type` tinyint(1) DEFAULT NULL COMMENT 'We are adding dm from which section. 1 = Endorsed Client, 2 = Endorsed Manager, 3 = Get endorsed from RS',
  `key_note` text,
  `headhunt_note` text,
  `created_user_id` int(11) DEFAULT NULL,
  `created_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `approved_date` datetime DEFAULT NULL,
  `last_mod_user_id` int(11) DEFAULT NULL,
  `mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `last_contacted_date` timestamp NULL DEFAULT NULL,
  `last_contacted_user_id` int(11) DEFAULT NULL,
  `last_contacted_company_id` int(11) DEFAULT NULL,
  `is_third_rep` tinyint(1) DEFAULT '0',
  `is_cb_migrated` tinyint(4) DEFAULT '0',
  `is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
  `eb_reff_id` int(11) DEFAULT NULL,
  `is_third_sync` tinyint(4) DEFAULT '0',
  `is_third_utm_sync` tinyint(4) DEFAULT '0',
  `merge_account_id` int(11) DEFAULT NULL,
  `is_linked_cron` int(1) DEFAULT '0',
  `is_dm_SC_invite` tinyint(2) NOT NULL DEFAULT '0',
  `indexed_error_flag_dm` tinyint(4) DEFAULT '0',
  `is_digital` tinyint(4) DEFAULT NULL COMMENT '0=''No'', 1="Yes''',
  `flag_missing_name` tinyint(2) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `endorsed_dm_status` (`endorsed_dm_status`),
  KEY `status` (`status`),
  KEY `mod_date` (`mod_date`),
  KEY `base_company_id` (`base_company_id`,`company_user_id`),
  KEY `fname` (`fname`),
  KEY `company_user_id` (`company_user_id`),
  KEY `base_user_id` (`base_user_id`),
  KEY `last_contacted_user_id` (`last_contacted_user_id`),
  KEY `lname` (`lname`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=555262 DEFAULT CHARSET=utf8 COMMENT='this stores the user or DM of company';

Answer 1

You should check your query first before going into optimization.

The GROUP_CONCAT -function is an aggregate function in SQL. This means that unless the GROUP_CONCAT is the only column you query, you should have the GROUP BY -clause in the query.

The GROUP BY -clause divides the query result into smaller sets into which then the aggregate function (in this case GROUP_CONCAT ) is applied to.

Older MySQL versions (and new ones configured the ONLY_FULL_GROUP_BY -mode off) allowed using aggregate function without GROUP BY -clause causing all sorts of errornous results.

To fix the issue, you should list the keys for those tables used in the selected columns ( GROUP BY ecu.id, ecuad.id, bc.id ).

After you have done this, check that you have indexs on column used in WHERE and the ones doing the JOIN s