Best way to bulk insert and update rows in multiple tables in mysql

Question

I have like 10 table inserts and each insert has 1 either 1000 or even 1 lakh records and have got 4 table updates as well. Basically for 1 lakh records, it would be 10 lakh inserts and 4 lakh updates and this could run on an hourly basis. And the main constraint is that the whole 10 table inserts and 4 table updates has to be in one transaction.

I read about several ways.For handling updates I might go with using a temporary table and and dump all the IDs which has to be updated to the table and then using a join with that temporary table from the REAL table to which the update has to be done. But I still dont have an effective solution for bulk inserting into 10 tables. It is eating up lot of time. LOAD_DATA file is an option, but that might not be a good choice provided in realtime I might end up creating 100s of thousands of files as this operation will happen across many databases and scopes. Please give your suggestions to increase the speed for these bulk inserts.

I am planning to change "bulk_insert_buffer_size" in mysql and any other parameters that would be useful as well. So any kind of suggestion would help me a lot here

Answer 1

How are you trying to do these bulk inserts? Are you using some client GUI like phpMyAdmin? Please try to run the SQL on the MySQL client prompt.

abhay> mysql -u<your_user> -h<your_host> -p < <path_to_your_sql_file> 

This should be fast enough to do all INSERT and UPDATES. Please do remember to put actual values where applicable. And the SQL file should contain valid INSERT and UPDATE statements.

Few suggestions:

1. you might try adding SET foreign_key_checks = 0 at the beginning of your SQL file and SET foreign_key_checks = 1 at the end
2. if your tables have many indexes, temporarily disabling them might help. Try ALTER TABLE...DISABLE KEYS and ALTER TABLE...ENABLE KEYS
3. You may even choose to split into multiple SQL files, one per table

Hope it helps!