Very slow MySQL LeftJoin on AWS RDS, but very quick elsewhere

Question

MySQL in PROD is running through AWS RDS (Aurora, db.t3.small). The resource ultization across the board is well within happy levels.

If I execute the following SELECT, it's almost instanteneous:

 SELECT SQL_CALC_FOUND_ROWS `submissions`.*
 FROM `cmsb_submissions` as `submissions`
 WHERE (status = '4') 
 LIMIT 15

If I add a LEFTJOIN, it takes upwards of 15 seconds:

SELECT SQL_CALC_FOUND_ROWS `submissions`.*,
`explorer_points_earning`.`num` AS `explorer_points_earning.num`,
`explorer_points_earning`.`createdDate` AS `explorer_points_earning.createdDate`,
`explorer_points_earning`.`createdByUserNum` AS `explorer_points_earning.createdByUserNum`,
`explorer_points_earning`.`updatedDate` AS `explorer_points_earning.updatedDate`,
`explorer_points_earning`.`updatedByUserNum` AS `explorer_points_earning.updatedByUserNum`,
`explorer_points_earning`.`user` AS `explorer_points_earning.user`,
`explorer_points_earning`.`points_earned` AS `explorer_points_earning.points_earned`,
`explorer_points_earning`.`for_trail` AS `explorer_points_earning.for_trail`
FROM `cmsb_submissions` as `submissions`
LEFT JOIN `cmsb_explorer_points_earning` AS `explorer_points_earning` 
     ON submissions.num = explorer_points_earning.for_trail
    AND explorer_points_earning.user = 7
WHERE (status = '4') 
 LIMIT 15

Neither table is large:

explorer_points_earning = 17,000 records
submissions = 1,000 records 

and the following INDEXES exist:

SHOW INDEX FROM cmsb_explorer_points_earning;

'cmsb_explorer_points_earning', '0', 'PRIMARY', '1', 'num', 'A', '17155', NULL, NULL, '', 'BTREE', '', ''
'cmsb_explorer_points_earning', '1', '_auto_for_trail', '1', 'for_trail', 'A', '2450', '16', NULL, 'YES', 'BTREE', '', ''
'cmsb_explorer_points_earning', '1', '_auto_user', '1', 'user', 'A', '3431', '16', NULL, 'YES', 'BTREE', '', ''

So, it's not a big query by any stretch.

The exact same query (with the LEFTJOIN), with the exact same database dump on my local development server (less resources) is instantaneous.

I'm confident it's not a RDS "resource" issue as I doubled the instance to a t3.medium and it made no difference.

Which leads me to think it's an error with the indexing, or some RDS specific configuration.

EXPLAIN:

ON RDS:

'1', 'SIMPLE', 'submissions', 'ALL', NULL, NULL, NULL, NULL, '1100', 'Using where'
'1', 'SIMPLE', 'explorer_points_earning', 'ALL', '_auto_for_trail', NULL, NULL, NULL, '17155', 'Range checked for each record (index map: 0x2)'

ON DEV:

'1', 'SIMPLE', 'submissions', NULL, 'ALL', NULL, NULL, NULL, NULL, '1064', '100.00', NULL
'1', 'SIMPLE', 'explorer_points_earning', NULL, 'ALL', '_auto_user,_auto_for_trail', NULL, NULL, NULL, '17082', '100.00', 'Using where; Using join buffer (hash join)'

Notable difference being RDS only has "auto_for_trail" as a 'possible key' whereas DEV has both fields, _auto_user& _auto_for_trail. And RDS is using a nested loop and DEV using hash join.

No doubt there's more info I can provide to help debug this, but I'm not sure what, so please let me know what else could help.

Any help, most appreciated,

Thanks

-- EXTRA INFO --

SHOW CREATE TABLE cmsb_explorer_points_earning;

RDS

CREATE TABLE 
`cmsb_explorer_points_earning` (
    `num` int(10) unsigned NOT NULL AUTO_INCREMENT, 
    `createdDate` datetime NOT NULL, 
    `createdByUserNum` int(10) unsigned NOT NULL, 
    `updatedDate` datetime NOT NULL, 
    `updatedByUserNum` int(10) unsigned NOT NULL, 
    `user` mediumtext COLLATE utf8mb4_unicode_ci, 
    `points_earned` mediumtext COLLATE utf8mb4_unicode_ci, 
    `for_trail` mediumtext COLLATE utf8mb4_unicode_ci, 
    PRIMARY KEY (`num`), 
    KEY `_auto_for_trail` (`for_trail`(16)), KEY `_auto_user` (`user`(16))
) ENGINE=InnoDB AUTO_INCREMENT=17191 DEFAULT 
CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci'

MySQL Version: 5.6.10

DEV/LOCAL

CREATE TABLE         
`cmsb_explorer_points_earning` (
    `num` int unsigned NOT NULL AUTO_INCREMENT, 
    `createdDate` datetime NOT NULL, 
    `createdByUserNum` int unsigned NOT NULL, 
    `updatedDate` datetime NOT NULL, 
    `updatedByUserNum` int unsigned NOT NULL, 
    `user` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, 
    `points_earned` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, 
    `for_trail` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, 
    PRIMARY KEY (`num`), 
    KEY `_auto_user` (`user`(16)), KEY `_auto_for_trail` (`for_trail`(16))
) ENGINE=InnoDB AUTO_INCREMENT=17181 DEFAULT 
CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci'

MySQL Version: 8.0.20-0ubuntu0.20.04.1

Answer 1

Add this:

INDEX(user, for_trail)

At that point, INDEX(user) is redundant and should be dropped.

If the "user" is a user name, you do not need MEDIUMTEXT (max of 16M bytes). Nor do you need TEXT (max of 64K bytes). Change it so something civilized like VARCHAR(100) .

Ditto for the other MEDIUMTEXTs , if practical. At that point, the index I am suggestion will work.

The other workaround is to use "prefixing" as you tried:

KEY `_auto_for_trail` (`for_trail`(16)),
KEY `_auto_user` (`user`(16))

But, you have already found out that that is rather useless.

Also, a LIMIT without an ORDER BY makes the choice of 15 rows unpredictable.