Slow query times on AWS RDS but not on DigitalOcean LAMP

Question

I have a database dump and MySQL query that is the same across three environments, but it is extremely slow on AWS, and I'd like to find out why.

Query runs 15s ~ 30s on AWS RDS but 8ms ~ 12ms on DigitalOcean LAMP. (Control experiment: Local MAMP stack runs 8s ~ 10s).

How is this possible that DigitalOcean LAMP stack is running the same query in ms?

Backstory: I migrated from DigitalOcean to AWS RDS. I am super shocked at the query speeds. I can't figure out why.

Note: Using T2.micro RDS instance. Have also tested with M4.2xLarge RDS instance. Query times are the same.

Here is the query:

SELECT `t`.`col1` AS `t0_c0`,
       `t`.`col2` AS `t0_c1`,
       `t`.`col3` AS `t0_c2`,
       `t`.`col4` AS `t0_c3`,
       `t`.`col5` AS `t0_c4`,
       `t`.`col6` AS `t0_c5`,
       `t`.`col7` AS `t0_c6`,
       `t`.`col8` AS `t0_c7`,
       `t`.`col9` AS `t0_c8`,
       `t`.`col10` AS `t0_c9`,
       `t`.`col11` AS `t0_c10`,
       `t`.`col12` AS `t0_c11`,
       `t`.`col13` AS `t0_c12`,
       `t`.`col14` AS `t0_c13`,
       `t`.`col15` AS `t0_c14`,
       `t`.`col16` AS `t0_c15`,
       `t`.`col17` AS `t0_c16`,
       `t`.`col18` AS `t0_c17`,
       `t`.`col19` AS `t0_c18`,
       `t`.`col20` AS `t0_c19`,
       `t`.`col21` AS `t0_c20`,
       `t`.`col22` AS `t0_c21`,
       `t`.`col23` AS `t0_c22`,
       `t`.`col24` AS `t0_c23`,
       `t`.`col25` AS `t0_c24`,
       `t`.`col26` AS `t0_c25`,
       `t`.`col27` AS `t0_c26`,
       `t`.`col28` AS `t0_c27`,
       `t`.`col29` AS `t0_c28`,
       `t`.`col30` AS `t0_c29`,
       `t`.`col31` AS `t0_c30`,
       `t`.`col32` AS `t0_c31`,
       `t`.`col33` AS `t0_c32`,
       `t`.`col34` AS `t0_c33`,
       `t`.`col35` AS `t0_c34`,
       `t`.`col36` AS `t0_c35`,
       `t`.`col37` AS `t0_c36`,
       `t`.`col38` AS `t0_c37`,
       `t`.`col39` AS `t0_c38`,
       `t`.`col40` AS `t0_c39`,
       `t`.`col41` AS `t0_c40`,
       `t`.`col42` AS `t0_c41`,
       `t`.`col43` AS `t0_c42`,
       `t`.`col44` AS `t0_c43`,
       `t`.`col45` AS `t0_c44`,
       `t`.`col46` AS `t0_c45`,
       `t`.`col47` AS `t0_c46`,
       `t`.`col48` AS `t0_c47`,
       `t`.`col49` AS `t0_c48`,
       COUNT(exactTagMatch.id) AS exactMatches,
       COUNT(tags.id) AS matches,
       `vendor`.`col1` AS `t1_c0`,
       `vendor`.`col2` AS `t1_c1`,
       `vendor`.`col3` AS `t1_c2`,
       `vendor`.`col4` AS `t1_c3`,
       `vendor`.`col5` AS `t1_c4`,
       `vendor`.`col6` AS `t1_c5`,
       `vendor`.`col7` AS `t1_c6`,
       `vendor`.`col8` AS `t1_c7`,
       `vendor`.`col9` AS `t1_c8`,
       `vendor`.`col10` AS `t1_c9`,
       `vendor`.`col11` AS `t1_c10`,
       `vendor`.`col12` AS `t1_c11`,
       `vendor`.`col13` AS `t1_c12`,
       `vendor`.`col14` AS `t1_c13`,
       `vendor`.`col15` AS `t1_c14`,
       `vendor`.`col16` AS `t1_c15`,
       `vendor`.`col17` AS `t1_c16`,
       `vendor`.`col18` AS `t1_c17`,
       `vendor`.`col19` AS `t1_c18`,
       `vendor`.`col20` AS `t1_c19`,
       `vendor`.`col21` AS `t1_c20`,
       `vendor`.`col22` AS `t1_c21`,
       `vendor`.`col23` AS `t1_c22`,
       `vendor`.`col24` AS `t1_c23`,
       `vendor`.`col25` AS `t1_c24`,
       `vendor`.`col26` AS `t1_c25`,
       `vendor`.`col27` AS `t1_c26`,
       `vendor`.`col28` AS `t1_c27`,
       `tags`.`col1` AS `t2_c0`,
       `tags`.`col2` AS `t2_c1`,
       `tags`.`col3` AS `t2_c2`,
       `tags`.`col4` AS `t2_c3`,
       `tags`.`col5` AS `t2_c4`,
       `tags`.`col6` AS `t2_c5`,
       `tags`.`col7` AS `t2_c6`,
       `tags`.`col8` AS `t2_c7`,
       `tags`.`col9` AS `t2_c8`,
       `tags`.`col10` AS `t2_c9`,
       `tags`.`col11` AS `t2_c10`,
       `tags`.`col12` AS `t2_c11`
FROM `table1` `t`
LEFT JOIN table3 AS exactTagMatch ON exactTagMatch.col1 = 1545
AND exactTagMatch.col1 = t.col1
LEFT OUTER JOIN `table2` `vendor` ON (`t`.`col1`=`vendor`.`id`)
LEFT OUTER JOIN `table3` `tags_tags` ON (`t`.`col1`=`tags_tags`.`col1`)
LEFT OUTER JOIN `table3` `tags` ON (`tags`.`id`=`tags_tags`.`col1`)
WHERE (((((t.col2 LIKE '%search query%')
          OR (vendor.col2 LIKE '%search query%'))
         OR (tags.col1 IN ('1386',
                         '1407',
                         '1408',
                         '1409',
                         '1410',
                         '1411',
                         '1416',
                         '1418',
                         '1512',
                         '1519',
                         '1528',
                         '1545',
                         '1583',
                         '1584',
                         '1585',
                         '1586',
                         '1604',
                         '1605')))
        AND (t.col2='active'))
       AND (t.col3='0'))
GROUP BY t.col1
ORDER BY exactMatches DESC,
         matches DESC,
         t.col5 DESC LIMIT 12;

MySQL Plan: AWS RDS (Slow)

+----+-------------+---------------+--------+---------------------------------+-------------------------+---------+-----------------------+-------+--------------------------------------------------------+
| id | select_type | table         | type   | possible_keys                   | key                     | key_len | ref                   | rows  | Extra                                                  |
+----+-------------+---------------+--------+---------------------------------+-------------------------+---------+-----------------------+-------+--------------------------------------------------------+
|  1 | SIMPLE      | t             | ref    | PRIMARY,activity_status_premium | activity_status_premium | 2       | const,const           |   594 | Using index condition; Using temporary; Using filesort |
|  1 | SIMPLE      | exactTagMatch | ALL    | NULL                            | NULL                    | NULL    | NULL                  | 24434 | Using where; Using join buffer (Block Nested Loop)     |
|  1 | SIMPLE      | vendor        | eq_ref | PRIMARY                         | PRIMARY                 | 4       | ebdb.t.vendor_id      |     1 | NULL                                                   |
|  1 | SIMPLE      | tags_tags     | ALL    | NULL                            | NULL                    | NULL    | NULL                  | 24434 | Using where; Using join buffer (Block Nested Loop)     |
|  1 | SIMPLE      | tags          | eq_ref | PRIMARY                         | PRIMARY                 | 4       | ebdb.tags_tags.tag_id |     1 | Using where                                            |
+----+-------------+---------------+--------+---------------------------------+-------------------------+---------+-----------------------+-------+--------------------------------------------------------+
5 rows in set (0.02 sec)

MySQL Plan: DigitalOcean (Fast)

+----+-------------+---------------+--------+-------------------------+-------------------------+---------+-------------------------------+-------+----------------------------------------------+
| id | select_type | table         | type   | possible_keys           | key                     | key_len | ref                           | rows  | Extra                                        |
+----+-------------+---------------+--------+-------------------------+-------------------------+---------+-------------------------------+-------+----------------------------------------------+
|  1 | SIMPLE      | t             | ref    | activity_status_premium | activity_status_premium | 2       | const,const                   |   590 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | exactTagMatch | ALL    | NULL                    | NULL                    | NULL    | NULL                          | 32473 |                                              |
|  1 | SIMPLE      | vendor        | eq_ref | PRIMARY                 | PRIMARY                 | 4       | db.t.vendor_id      |     1 |                                              |
|  1 | SIMPLE      | tags_tags     | ALL    | NULL                    | NULL                    | NULL    | NULL                          | 32473 |                                              |
|  1 | SIMPLE      | tags          | eq_ref | PRIMARY                 | PRIMARY                 | 4       | db.tags_tags.tag_id |     1 | Using where                                  |
+----+-------------+---------------+--------+-------------------------+-------------------------+---------+-------------------------------+-------+----------------------------------------------+
5 rows in set (0.01 sec)

---

Here is a comparison between the MySQL versions:

MySQL Version: AWS RDS (Slow)

mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.6.27                       |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| version                 | 5.6.27                   |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
7 rows in set (0.03 sec)

MySQL Version: DigitalOcean (Fast)

mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name           | Value                   |
+-------------------------+-------------------------+
| innodb_version          | 5.5.44                  |
| protocol_version        | 10                      |
| slave_type_conversions  |                         |
| version                 | 5.5.44-0ubuntu0.14.04.1 |
| version_comment         | (Ubuntu)                |
| version_compile_machine | x86_64                  |
| version_compile_os      | debian-linux-gnu        |
+-------------------------+-------------------------+
7 rows in set (0.01 sec)

---

Here is a comparison between the .cnf files: https://www.diffnow.com/?report=ze4m0

Note: Had to use external link because body exceeded character count

---

Update

Just ran more tests. On DigitalOcean, the first time i run the query it took 14s. Thereafter it runs in in ms. I believe there is some sort of caching/indexing going on there. Would like to find out what could result in this.

Answer 1

In MySql 5.6+, the query cache is disabled by default. In your my.cnf files you can see that by looking for "query_cache_type" - you'll see it's OFF on RDS, but not on the DO db.

I'd bet that if you run the query for the first time on DO it will be a similar speed - no other conf change is likely to result in such dramatic differences.

As the cache is off now by default, you should read up on the reasons for that change - it's your call if you need it or not, but just note that it was disabled deliberately!