使用PDO向MySQL插入1000行的有效方法

Question

I have to insert 1000 rows of data per time into MySQL. At the moment, I use PDO and for-loop to insert row by row to database. Is there any more efficient way to achieve better performance? Because I have to set max_execution_time to 5 minutes.

function save()
{
            return $query = $this->insert("
                INSERT INTO gadata (landing_page, page_title, page_views, visits, visitors, bounce_rate, pageviews_per_visit, time_on_page, avg_time_on_page, day, month, year, hour)
                VALUES (:landing_page, :page_title, :page_views, :visits, :visitors, :bounce_rate, :pageviews_per_visit, :time_on_page, :avg_time_on_page, :day, :month, :year, :hour)", $this->data);
}

And

protected function insert($sql, array $data) {
    $q = $this->_db_handler->prepare($sql);

    foreach ($data as $k => $v)
    {
       $q->bindValue(':' . $k, $v);
    }

    $q->execute();
}

Answer 1

It is not PDO nor the way you are inserting makes insert so delayed, but innodb engine. So you have 3 choices:

1. Wrap all inserts into transaction.
2. using root privileges, set innodb_flush_log_at_trx_commit variable to 2, to make innodb use a filecache for writes - it will make your inserts blazingly fast.
3. Run all the inserts in one query as suggested by Manu

Answer 2

可能不是最好的解决方案，但是您可以尝试构造一个查询字符串，例如INSERT INTO [table] VALUES (r1c1,r1c2,r1c3),(r2c1,r2c2,r2c3) ...并执行一个mysql_query （或者说一个查询几百行），甚至在构建sql查询时，如果不是来自受信任的来源，您甚至可以通过编程方式验证数据。

Answer 3

Parameterized queries by definition trade execution safety against reduced flexibility on the count of data items.

You have at least 2 possibilities to mitigate:

• Build up the SQL, then execute it at once:

Something like:

$sql="INSERT INTO gadata (landing_page, page_title, page_views, visits, visitors, bounce_rate, pageviews_per_visit, time_on_page, avg_time_on_page, day, month, year, hour) VALUES ";
foreach ($all_data_rows as $data) {
  if ($i==0) $value=""; else $value=",";
  $sql.=$value."(:landing_page$i, :page_title$i, :page_views$i, :visits$i, :visitors$i, :bounce_rate$i, :pageviews_per_visit$i, :time_on_page$i, :avg_time_on_page$i, :day$i, :month$i, :year$i, :hour$i)";
  $i++;
}
$i=0;
$q=$db_handler->prepare($sql);
foreach ($all_data_rows as $data) {
  foreach ($data as $k => $v) {
    $q->bindValue(":$k$i", $v);
  }
  $i++;
}
$q->execute();

• Use a temporary table to avoid locking and disk overhead

First create a temporary table of type HEAP with the same structure as your target table, then insert into it: This will be much faster, as no locking and disk IO happens. Then run

INSERT INTO final_table SELECT * FROM temporary_table

If mitigation doesn't suffice, you will need to consider using non-parameterized queries for this use case. The usual caveats apply.