如何使用php向MySQL中插入/更新大量数据

Question

I have an excel sheet which has a large amount of data. I am using php to insert the data into mysql server.

I have two problems

1) I have to update a row if the id already exists, else insert the data.

2) BIG PROBLEM : I have more than 40,000 rows and the time out on the sql server which is set by the admin is 60 seconds. When i run the update/insert query it will take more than 60 seconds, and because of this there will be a timeout. So the whole process will fail.

Is there a way I can do this ?

Currently I am checking the student id if it exists, then update otherwise insert. This I feel is taking a lot of time and causing the server to time out.

Also I have this field in the mysql stating the last time the data was updated(last_update). I was thinking of using this date, and if it is past a particular date(ie last time i ran the program) then only those rows should be updated.

Will this help in anyway ?

And what is the query i can run so as to check this date in the mysql database, that if it is past a particular date only those rows need to be updated and not everything else. (Please help me with an example query for the above!!!!!!!!!!!!!!!!!)

Answer 1

Assuming that you are using InnoDB engine (which is default in most recent MySQL versions), you should simply use transactions: wrap your insert loop into BEGIN; ... COMMIT; block.

By default, every statement is run as transaction, and server must make sure that data makes it safely to disk before continuing to next statement. If you start transaction, then do many inserts, and then commit transaction, only then server must flush all the data onto the disk. On modern hardware, this could amount only to few disk operations instead of 500k of them. Another consideration is to use prepared statements . Server has to parse every SQL statement before executing it. This parsing does not come for free, and it is not unusual that parsing time could be more expensive than actual query execution time. Usually, this parsing is done every single time, and for your case it is done 500k times. If you use prepared statements, parsing/preparation is done only once, and cost to execute statement is only disk write (which is enhanced further if you are within active transaction, because server can batch that write by delaying it until transaction commits).

Total improvement from using these methods could be dramatic - I have seen situations when using transactions reduced total run time from 30 minutes to 20 seconds.

http://php.net/manual/en/pdo.prepared-statements.php