PHP Mysql多次插入（大数据集）

Question

I have been running a relatively simple script, however it tends to lock up the database.

Each time a lookup value is inserted, it is also checked against to make sure it isnt inserted again.

This works very well on small data sets (<50k), however it has issues with large data sets (>2m). Any help would be appreciated.

$insertCounter = 20;
$matchCounter = 200;

$insertIndex = 0;
$sqlInsert = 'INSERT INTO `database` (`lookup_value`, `timestamp`, `source`) VALUES ';
$matchIndex = 0;
$resetCount = 0;
$indexCounter = 0;

foreach ($matches as $lookup) {
    $sqlSelect = 'SELECT `id` FROM `database` WHERE `lookup_value` = \'' . $lookup . '\'';
    $qExisting = ExecuteSQL($sqlSelect);

    if (mysql_num_rows($qExisting) == 0) {
        $insertIndex += 1;
        $sqlInsert .= '(\'' . strtolower($lookup) . '\', \'' . date('Y-m-d H:i:s') . '\', \'database\'), ';

        if ($insertIndex >= $insertCounter) {
            $sqlInsert = substr($sqlInsert, 0, strlen($sqlInsert) - 2);
            ExecuteSQLNoResult($sqlInsert);     
            echo '<p><strong>' . date('Y-m-d H:i:s') . '</strong><br />' . $sqlInsert . '</p>';

            $sqlInsert = 'INSERT INTO `database` (`lookup_value`, `timestamp`, `source`) VALUES ';
            $insertIndex = 0;
        }
    }

    mysql_free_result($qExisting);
    $matchIndex += 1;
    $indexCounter += 1;

    if ($matchIndex > $matchCounter) {
        $resetCount += 1;
        $matchIndex = 0;
        echo '<p>(' . $indexCounter . ', reset no.' . $resetCount . ') Counter Reached, resetting.</p>';
    }
}

Answer 1

如何添加带有lookup_value字段的UNIQUE索引，然后使用INSERT IGNORE语句？