mysql query slow after MyISAM -> InnoDB and move to AWS

Question

I plan on switching from a shared hosted server to an AWS setup (EC2 for apache/php, RDS for MySQL).

I had a beta version of my php/mysql site running on the shared server for over a year. One page in particular always ran fast (probably < 3 seconds to completely load) in browser.

In preparing for the move, I changed the database and all tables from MyISAM to InnoDB. I also switched the default character set to utf8mb4 (was latin-1 or something) and collation to utf8mb4_unicode_ci. There were a couple of columns that were something like varchar2(400) that were indexed that got the > 191 character error (because of the move from 3byte->4byte with utf8mb4). I went in and manually switched just those columns to UTF-8/utf_unicode_ci. EVerything seems ok (no explicit errors)

Now, I've moved a copy of the site/DB to AWS. Most pages load fine. THis one in particular is taking forever to load (> 5 mins). I have some pretty small EC2 instances and RDS instances so I realize that could be an issue, but I'm the only one using the site (for one query) and the entire database is < 70MB.

Running an explain plan shows in most cases keys (indexes?) are used and one "derived" table that has 74k rows. 74k is a very small dataset. I'm much more familiar with Oracle explain plans, so it is difficult for me to figure out what is going on.

I tried running the original page on the shared hosting server and it has gotten much slower too! Therefore, I have no reason to believe it's an AWS issue.

I know that MyISAM is a little faster in simple scenarios, but there is no way it should be 10000000000000x faster than my new use of InnoDB?

Did something go wrong with the conversion where it was trying to index varchars with > 191 characters? is it possible it just somehow broke the system/tables? The queries will finish if you give them a ton of time, but there's no way they should be this slow. I would think that even if indexes were fried and it was doing full table scans on 74k rows it shouldn't even break a sweat.

Thoughts?

Edit: Found this thread: https://dba.stackexchange.com/questions/75091/why-are-simple-selects-on-innodb-100x-slower-than-on-myisam The poster seems to have a similar slowdown experience. I can't use the same solution as him/her. It's insane to think I'd have to put all columns that I want to return into an index. There is no way this is something everyone just accepts?

Answer 1

Ok, I figured it out. Was my fault (not surprising). I wrote the query a long time ago before I really knew what I was doing. There was definitely room for improvement...

Query looked something like:

TableA (main table of focus)
  aID (pk)
  bID (fk)
  locationID (fk)
  rowTitle

TableB 
  bID (pk)
  locationID (fk)

TableLocations
  locationID (pk)
  locationName

I wanted to basically use A's location if it existed, but fall back to B's location if A didn't have one. (A would always be linked to B).

I was doing some like

 SELECT
   rowTitle,
   locationName
 FROM
   TableA a, TableB b, TableLocations loc
 WHERE
  a.bID = b.bID
  AND
  ( a.locationID = loc.locationID
    OR 
   (a.locationID IS NULL AND b.locationID = loc.locationID)
  )

It worked fine and always returned what I wanted when using MyISAM. And it was very quick. However, the sh$& hit the wall when I moved over to InnoDB for some reason. No clue on how the engines differ when it comes to executing the plans for this query. I imagine at some point MyISAM would have choked when the dataset got large enough, but since I'm still in dev/uat the data is small.

I've since moved to something structurally better like:

 SELECT 
   rowTitle
   CASE WHEN loc1.locationID IS NOT NULL THEN loc1.locationName
        WHEN loc2.locationID IS NOT NULL THEN loc2.locationName
   END as locationName
FROM
   TableA a
JOIN TableB b ON a.bID = b.bID
 LEFT OUTER JOIN TableLocations loc1 ON a.locationID = loc1.locationID
 LEFT OUTER JOIN TableLocations loc2 ON b.locationID = loc2.locationID

And everything works great! I cut out a lot of the query and other tables to focus on my main logical/physical problem.

The queries are down to about 10MS which is what I expect.

That's the one thing I love and hate about MySQL. Sometimes "it just works" and let's you look past your own deficiencies. Definitely my fault here for not doing it right the first time...

Additional Question: If anyone knows the difference between how MyISAM vs InnoDB would have evaluated the first (bad) query and why the performance was so different, i'd really love to hear it!

Thanks!