MySQL Order By Very Slow

Question

I understand there are a lot of Order By Questions with various solutions, but what I'm asking for is specific to the data structure that I have setup below.

I have the following 2 tables setup:

Table Setup

CREATE TABLE `record` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `file_group_id` INT(11) NOT NULL,
  `status_id` INT(11) NOT NULL,
  `title` VARCHAR(3000) NOT NULL,
  `date_created` DATETIME NOT NULL,
  `user_created` INT(11) NOT NULL,
  `publish_date` DATETIME NOT NULL,
  PRIMARY KEY (`id`),
  KEY `IDX_RECORD_FGID` (`file_group_id`)
);

CREATE TABLE `record_meta_text` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `record_id` INT(11) NOT NULL,
  `column_id` INT(11) NOT NULL,
  `value` VARCHAR(3000) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `IDX_RMT_VALUE` (`value`(800)),
  KEY `IDX_RMT_COL_ID` (`column_id`),
  KEY `IDX_RMT_RECORD_ID_COL_ID` (`record_id`,`column_id`,`value`(800))
)

Query without Order By

SELECT
    r.title AS col_title,
    rmt_2780.value AS col_2780,
    rmt_2781.value AS col_2781,
    rmt_18474.value AS col_18474
FROM 
    record r
INNER JOIN record_meta_text AS rmt_2780 ON rmt_2780.record_id = r.id AND rmt_2780.column_id = 2780
INNER JOIN record_meta_text AS rmt_2781 ON rmt_2781.record_id = r.id AND rmt_2781.column_id = 2781
INNER JOIN record_meta_text AS rmt_18474 ON rmt_18474.record_id = r.id AND rmt_18474.column_id = 18474
WHERE 
    r.file_group_id = 2350 
    AND r.status_id = 1
LIMIT 0, 50

Output and Explain

Execution Time: .004 seconds

Here is the EXPLAIN output:

    id  select_type  table      type    possible_keys                            key                       key_len  ref                             rows  Extra        
------  -----------  ---------  ------  ---------------------------------------  ------------------------  -------  ----------------------------  ------  -------------
     1  SIMPLE       r          ref     PRIMARY,IDX_RECORD_FGID                  IDX_RECORD_FGID           4        const                         527895  Using where  
     1  SIMPLE       rmt_18474  ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL)       
     1  SIMPLE       rmt_2780   ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL)       
     1  SIMPLE       rmt_2781   ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL) 

Query with Order By

SELECT
    r.title AS col_title,
    rmt_2780.value AS col_2780,
    rmt_2781.value AS col_2781,
    rmt_18474.value AS col_18474
FROM 
    record r
INNER JOIN record_meta_text AS rmt_2780 ON rmt_2780.record_id = r.id AND rmt_2780.column_id = 2780
INNER JOIN record_meta_text AS rmt_2781 ON rmt_2781.record_id = r.id AND rmt_2781.column_id = 2781
INNER JOIN record_meta_text AS rmt_18474 ON rmt_18474.record_id = r.id AND rmt_18474.column_id = 18474
WHERE 
    r.file_group_id = 2350 
    AND r.status_id = 1
ORDER BY col_2780
LIMIT 0, 50 

Output and Explain

Execution Time: 35.237 seconds

    id  select_type  table      type    possible_keys                            key                       key_len  ref                             rows  Extra                                         
------  -----------  ---------  ------  ---------------------------------------  ------------------------  -------  ----------------------------  ------  ----------------------------------------------
     1  SIMPLE       r          ref     PRIMARY,IDX_RECORD_FGID                  IDX_RECORD_FGID           4        const                         527895  Using where; Using temporary; Using filesort  
     1  SIMPLE       rmt_2780   ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL)                                        
     1  SIMPLE       rmt_2781   ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL)                                        
     1  SIMPLE       rmt_18474  ref     IDX_RMT_COL_ID,IDX_RMT_RECORD_ID_COL_ID  IDX_RMT_RECORD_ID_COL_ID  8        file_cabinet_data.r.id,const       1  (NULL)      

Question and Solution

So, my question is, how can I get this to not take forever with the order by. I will have in several cases multiple order by conditions, as well as where statements. The LIMIT/OFFSET is for paging.

The record table currently has 1,139,119 records.

The file_meta_text table currently has 7,584,428 records.

I have a SQLFiddle with the base schema and query, but no data added (too many records):

http://sqlfiddle.com/#!2/6ffc3/1

Any help would be greatly appreciated.

Answer 1

Can I suggest to use (because of the where clause):

KEY `IDX_RECORD_FGID` (`file_group_id`,`status_id`)

Instead of:

KEY `IDX_RECORD_FGID` (`file_group_id`)

And (because of complete indexing)

 KEY  `IDX_RMT_VALUE` (`value`)

Instead of:

KEY  `IDX_RMT_VALUE` (`value`(800))

But remember that after the 900bytes mysql doesn't index, that is the problem about order by, that is a mysql action that use a field not completely indexed. Are you sure you need to have the "value" field with length 3000?

And I think that you can have the same information using this cleaner query:

SELECT
    r.title AS col_title,
    rmt.value AS value,
    rmt.column_id AS column
FROM 
    record r
INNER JOIN record_meta_text AS rmt ON rmt.record_id = r.id
WHERE 
    rmt.column_id IN(2780,2781,18474)
    AND
    r.file_group_id = 2350 
    AND r.status_id = 1
LIMIT 0, 50