我如何加快除循环之外的mysql更新

Question

I have to read 460,000 records from one database and update those records in another database. Currently, I read all of the records in (select * from...) and then loop through them sending an update command to the second database for each record. This process is slower than I hoped and I was wondering if there is a faster way. I match up the records by the one column that is indexed (primary key) in the table.

Thanks.

Answer 1

我可能会优化获取大小进行读取（如setFetchSize(250)和JDBC -批量处理的写入（例如250个记录的批量大小）。

Answer 2

1. USE MULTI QUERY

aha. 'another db' means remote database.. in this case you SHOULD reduce number of interaction with remote DB. I suggest that use MULTIPLE QUERY . eg to execute 1,000 UPDATE at once,

$cnt = 1;
for ($row in $rows)
{
    $multi_query .= "UPDATE ..;";

    if ($cnt % 1000 == 0)
    {
        mysql_query($multi_query);
        $cnt = 0;
        $multi_query = "";
    }
    ++$cnt;

}

Normally Multi query feature is disable (for security reason), To use Multi query

• PHP : http://www.php.net/manual/en/mysqli.quickstart.multiple-statement.php
• C API : http://dev.mysql.com/doc/refman/5.0/en/c-api-multiple-queries.html
• VB : http://www.devart.com/dotconnect/mysql/docs/MultiQuery.html (I'm not a VB user, so not sure this is for MULTI Query for VB)

2. USE Prepared Statement

(When you are already using prepared stmt. skip this)

You are running 460K same structured Queries. So If you use PREPARED STATEMENT , you can obtain two advantages.

• Reduce query compile time
  • without prepared stmt. All queries are compiled, but just one time with prepared stmt.
• Reduce Network Cost
  • Assuming each UPDATE query is 100 bytes long, and there are 4 parameters (each is 4 bytes long)
  • without prepare stmt : 100 bytes * 460K = 46M
  • with prepare stmt : 16 bytes * 460K = 7.3M
  • it doesn't reduce dramatically

Here is how to use prepared statement in VB.

Answer 3

I am assuming your "other database" is on a separate server, so can't just be directly joined.

The key is to have fewer update statements. It can often be faster to insert your data into a new table like this:

create table updatevalues ( id int(11), a int(11), b int(11), c int(11) );
insert into updatevalues (id,a,b,c) values (1,1,2,3),(2,4,5,6),(3,7,8,9),...
update updatevalues u inner join targettable t using (id) set t.a=u.a,t.b=u.b,t.c=u.c;
drop table updatevalues;

(batching the inserts into however many statements you can fit in however big your maximum size is configured at, usually in the megabytes).

Alternatively, find unique values and update them together:

update targettable set a=42 where id in (1,3,7);
update targettable set a=97 where id in (2,5);
...
update targettable set b=1 where id in (1,7);
...

Answer 4

What I ended up doing was using a loop to concatenate my queries together. So instead of sending one query at a time, I would send a group at a time separated by semicolons:

update sometable set x=1 where y =2; update sometable set x = 5 where y = 6; etc...

This ended up improving my time by about 40%. My update went from 3 min 23 secs to 2 min 1 second.

But there was a threshold, where concatenating too many together started to slow it down again when the string got too long. So I had to tweak it until I found just the right mix. It ended up being 100 strings concatenated together that gave the best performance.

Thanks for the responses.