Mysql Insert into with select from myisam to innodb

Question

I have a table, which is accessed by multiple worker applications, which grab a batch of records, calculate some stuff on them and update the calculated result back into this table. This table is currently MyISAM based, but since we had some changes to our calculating algorithm and it resulted in a huge speed up, we have seen loads of locks on the table, which is expected as it's a MyISAM table. (The table has around 650-700 000 records in it).

Once all the records are processed, a report is generated based on it and the table is truncated. Once the table is truncated, a processes reinitalizeses it and the cycle starts again.

I was thinking about switching over the MyISAM table to InnoDB to prevent locks, but the processes, which initalizes the table with a INSERT INTO .. SELECT .. FROM statement (which used to take 3-4 mintues), was running for 35-40 minutes when i decided to stop it. Why is this? Is there a way to speed it up?

Answer 1

MyISAM stores records in the order they're inserted. On the other hand, InnoDB's primary key is a clustered index, so records are physically stored in the same order as the primary key. Therefore, ensure you're inserting in primary key order. You can add an ORDER BY clause to the INSERT INTO ... SELECT.

Also, to avoid split pages and rebuilding of your secondary indexes, add your secondary indexes after you've inserted the initial rows. So, remove the indexes, then re-add them after.

Also, to avoid the row locking (and auto-increment locking) overhead, explicitly lock the tables with LOCK TABLES.

Finally, increase innodb_buffer_pool_size , otherwise your indexes will build very slowly. The default is only 8MB.