why select different column can affect query speed in mysql 5.6?

Question

I have a table cdc_bond_valuation in mysql5.6:

+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+
| table_schema | table_name         | index_schema | index_name                      | seq_in_index | column_name  | cardinality |
+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+
| ss_product   | cdc_bond_valuation | ss_product   | IDX_cdc_bond_valuation_Bond_Key |            1 | Bond_Key     |      377844 |
| ss_product   | cdc_bond_valuation | ss_product   | IndexValuateDate                |            1 | Valuate_Date |      143025 |
| ss_product   | cdc_bond_valuation | ss_product   | PRIMARY                         |            1 | ID           |    25315548 |
+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+



query 1:
SELECT  Valuate_Date  FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;

query 2:
SELECT ID, Valuate_Date  FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;

query 3:
SELECT  Bond_Key FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;

query 4:
SELECT  Bond_Key,Valuate_Date FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;


explain 1:
mysql> explain SELECT  Valuate_Date  FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
| id | select_type | table              | type | possible_keys    | key              | key_len | ref   | rows  | Extra       |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
|  1 | SIMPLE      | cdc_bond_valuation | ref  | IndexValuateDate | IndexValuateDate | 5       | const | 98156 | Using index |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
1 row in set

explain 2:
mysql> explain SELECT  ID,Valuate_Date FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
| id | select_type | table              | type | possible_keys    | key              | key_len | ref   | rows  | Extra       |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
|  1 | SIMPLE      | cdc_bond_valuation | ref  | IndexValuateDate | IndexValuateDate | 5       | const | 98156 | Using index |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------------+
1 row in set


explain 3:
mysql> explain SELECT  Bond_Key FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
| id | select_type | table              | type | possible_keys    | key              | key_len | ref   | rows  | Extra |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
|  1 | SIMPLE      | cdc_bond_valuation | ref  | IndexValuateDate | IndexValuateDate | 5       | const | 98156 | NULL  |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
1 row in set

explain 4:
mysql> explain SELECT  Bond_Key,Valuate_Date FROM cdc_bond_valuation WHERE Valuate_Date = 20181203;
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
| id | select_type | table              | type | possible_keys    | key              | key_len | ref   | rows  | Extra |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
|  1 | SIMPLE      | cdc_bond_valuation | ref  | IndexValuateDate | IndexValuateDate | 5       | const | 98156 | NULL  |
+----+-------------+--------------------+------+------------------+------------------+---------+-------+-------+-------+
1 row in set



mysql> select table_schema,
       table_name,
       index_schema,
       index_name,
       seq_in_index,
       column_name,
       cardinality
  from information_schema.statistics
 where table_name = 'cdc_bond_valuation'
 order by table_schema, table_name, index_name, seq_in_index;

+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+
| table_schema | table_name         | index_schema | index_name                      | seq_in_index | column_name  | cardinality |
+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+
| ss_product   | cdc_bond_valuation | ss_product   | IDX_cdc_bond_valuation_Bond_Key |            1 | Bond_Key     |      377844 |
| ss_product   | cdc_bond_valuation | ss_product   | IndexValuateDate                |            1 | Valuate_Date |      143025 |
| ss_product   | cdc_bond_valuation | ss_product   | PRIMARY                         |            1 | ID           |    25315548 |
+--------------+--------------------+--------------+---------------------------------+--------------+--------------+-------------+
3 rows in set

mysql> 

there are 4 querys on the table as above, they all use index IndexValuateDate, but query 1 and 2 are very fast(less than 1 second), but query 3 and 4 very slow(more than 1000 seconds).

I notice that 1 and 2 just using index to feedback query(ID is primary key and Valuate_Date is indexed) . 3 and 4 firstly use index on Valuate_Date to filter table and then back to table get column with rowid ? why not just use index like 1 and 2 , since Bond_Key also indexed?

Answer 1

Please provide SHOW CREATE TABLE .

InnoDB silently adds the column(s) of the PRIMARY KEY to each secondary key. Hence queries 1 and 2 perform identically. They use just the index. This is indicated in EXPLAIN by Using index . That is, INDEX(Valuate_Date) contains the needed columns, and the other columns are not needed.

The EXPLAINs indicate that the same index was used, but it was not "covering" (no mention of Using index ). So the index was scanned linearly, but for each of the estimated 98156 entries with that date, it had to look up (in the data's BTree) the value of Bond_Key . This extra lookup caused the severe slowdown. (The 1000 seconds matches quite well with doing 98156 disk hits on an HDD.)

To make all 4 queries fast, replace IndexValuateDate with this composite index, and put the columns in the order given:

INDEX(Valuate_Date, Bond_Key, ID)

May I suggest that you work with dates via the DATE datatype, not DECIMAL(8,0) .

Unlike other databases, MySQL has no "rowid". Instead the PRIMARY KEY is used in the BTree for ordering the data.

Answer 2

    CREATE TABLE `cdc_bond_valuation` (
  `ID` varchar(32) NOT NULL,
  `Bond_Key` varchar(25) DEFAULT NULL,
  `Short_Name` varchar(32) DEFAULT NULL,
  `Bond_ID` varchar(32) DEFAULT NULL,
  `Valuate_Date` decimal(8,0) DEFAULT NULL,
  `Listed_Market` varchar(3) DEFAULT NULL,
  `Remaining_Year` decimal(7,4) DEFAULT NULL,
  `Val_Intraday_Dirty_Price` decimal(7,4) DEFAULT NULL,
  `Val_Intraday_Accrued_Interest` decimal(7,4) DEFAULT NULL,
  `Val_Clean_Price` decimal(7,4) DEFAULT NULL,
  `Val_Yield` decimal(7,4) DEFAULT NULL,
  `Val_Modified_Duration` decimal(7,4) DEFAULT NULL,
  `Val_Convexity` decimal(7,4) DEFAULT NULL,
  `Val_Basis_Point_Value` decimal(7,4) DEFAULT NULL,
  `Val_Spread_Duration` decimal(7,4) DEFAULT NULL,
  `Val_Spread_Convexity` decimal(7,4) DEFAULT NULL,
  `Market_Dirty_Price` decimal(7,4) DEFAULT NULL,
  `Market_Clean_Price` decimal(7,4) DEFAULT NULL,
  `Market_Yield` decimal(7,4) DEFAULT NULL,
  `Market_Modified_Duration` decimal(7,4) DEFAULT NULL,
  `Market_Convexity` decimal(7,4) DEFAULT NULL,
  `Market_Basis_Point_Value` decimal(7,4) DEFAULT NULL,
  `Market_Spread_Duration` decimal(7,4) DEFAULT NULL,
  `Market_Spread_Convexity` decimal(7,4) DEFAULT NULL,
  `Credibility` varchar(16) DEFAULT NULL,
  `Val_Rate_Duration` decimal(7,4) DEFAULT NULL,
  `Val_Rate_Convexity` decimal(7,4) DEFAULT NULL,
  `Market_Rate_Duration` decimal(7,4) DEFAULT NULL,
  `Market_Rate_Convexity` decimal(7,4) DEFAULT NULL,
  `Val_Closed_Dirty_Price` decimal(7,4) DEFAULT NULL,
  `Val_Closed_Accrued_Interest` decimal(7,4) DEFAULT NULL,
  `Remaining_Par_Value` decimal(7,4) DEFAULT NULL,
  `Val_Spread` decimal(7,4) DEFAULT NULL,
  `Yield_Curve_ID` varchar(128) DEFAULT NULL,
  `Market_Spread` decimal(7,4) DEFAULT NULL,
  `Absolute_Liquidity_Coefficient` decimal(7,4) DEFAULT NULL,
  `Position_Percentage` decimal(7,4) DEFAULT NULL,
  `Relative_Liquidity_Coefficient` decimal(7,4) DEFAULT NULL,
  `Relative_Liquidity_Value` decimal(7,4) DEFAULT NULL,
  `Option` varchar(8) DEFAULT NULL ,
  PRIMARY KEY (`ID`),
  KEY `IndexValuateDate` (`Valuate_Date`) USING BTREE,
  KEY `ValuateDateBondKey` (`Valuate_Date`,`Bond_Key`),
  KEY `IndexBondKey` (`Bond_Key`,`Listed_Market`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Answer 3

Query 5:

mysql> explain SELECT Bond_Key, Valuate_Date  FROM cdc_bond_valuation WHERE Valuate_Date = 20181203  and bond_key='C0000832017CORLEB01';
+----+-------------+--------------------+------+--------------------------------------------------+---------------------------------+---------+-------+------+------------------------------------+
| id | select_type | table              | type | possible_keys                                    | key                             | key_len | ref   | rows | Extra                              |
+----+-------------+--------------------+------+--------------------------------------------------+---------------------------------+---------+-------+------+------------------------------------+
|  1 | SIMPLE      | cdc_bond_valuation | ref  | IndexValuateDate,IDX_cdc_bond_valuation_Bond_Key | IDX_cdc_bond_valuation_Bond_Key | 78      | const |  787 | Using index condition; Using where |
+----+-------------+--------------------+------+--------------------------------------------------+---------------------------------+---------+-------+------+------------------------------------+
1 row in set

mysql> 

Query 5 uses index(bond_key) to filter, and then scan the result (787 rows) like full table scann for the bond_key . Query 5 doesn't use index(ValuateDate) at all. Is that right?

Answer 4

SELECT  count(*)  FROM cdc_bond_valuation WHERE   bond_key='C0000832017CORLEB01';
SELECT  count(*)  FROM cdc_bond_valuation WHERE   bond_key='C0000832017CORLEB01' and Valuate_Date = 20181203;


+----------+
| count(*) |
+----------+
|      788 |
+----------+
1 row in set

+----------+
| count(*) |
+----------+
|        2 |
+----------+
1 row in set

mysql>