MySQL不会插入所有成功的插入查询…为什么？

Question

Before I go on, this is purely a question of intuition. That is, I'm not seeking answers to work out specific bugs in my PHP/MySQL code. Rather, I want to understand what the range of possible issues that I need to consider in resolving my issue. To these ends, I will not post code or attach scripts - I will simply explain what I did and what is happening.

I have written PHP script that

1. Reads a CSV text file of X records to be inserted into a MySQL database table and/or update duplicate entries where applicable;
2. Inserts said records into what I will call a "root" table for that data set;
3. Selects subset records of specific fields from the "root" table and then inserts those records into a "master" table; and
4. Creates an output export text file from the master table for distribution.

There are several CSV files that I am processing via separate scheduled cron tasks every 30 minutes. All said, from the various sources, there are an estimated 420,000 insert transactions from file to root table, and another 420,000 insert transactions from root table to master table via the scheduled tasks.

One of the tasks involves a CSV file of about 400,000 records by itself. The processing contains no errors, but here's the problem: of the 400,000 records that MySQL indicates have been successfully inserted into the root table, only about 92,000 of those records actually store in the root table - I'm losing about 308,000 records from that scheduled task.

The other scheduled tasks process about 16,000 and 1,000 transactions respectively, and these transactions process perfectly. In fact, if I reduce the number of transactions from 400,000 to, say, 10,000, then these process just fine as well. Clearly, that's not the goal here.

To address this issue, I have tried several remedies...

1. Upping the memory of my server (and increasing the max limit in the php.ini file)
2. Getting a dedicated database with expanded memory (as opposed to a shared VPS database)
3. Rewriting my code to substantially eliminate stored arrays that suck down memory and process fgetcsv() processes on the run
4. Use INSERT DELAYED MySQL statements (as opposed to plain INSERT statements)

...and none of these remedies have worked as desired.

What range of remedial actions should be considered at this point, given the lack of success in the actions taken so far? Thanks...

Answer 1

The source data in csv may have duplicate records. Even though there are 400,000 record in the csv, your 'insert or update' logic trims them into reduced set. Less memory could lead to exceptions etc, but this kind of data loss.

Answer 2

I suspect there are problems in the CSV file.

My suggestion:

• Print something for debugging information on each lines read from CSV. This will show you how many lines are processed.
• On every insert/update, print any error (if any)

It's something like this:

<?php
$csv = fopen('sample.csv', 'r'); $line = 1;
while (($item = fgetcsv($csv)) !== false) {
    echo 'Line ' . $line++ . '... ';

    $sql = ''; // your SQL query
    mysql_query($sql);
    $error = mysql_error();

    if ($error == '') {
        echo 'OK' . PHP_EOL;
    } else {
        echo 'FAILED' . PHP_EOL . $error . PHP_EOL;
    }
}

So, if there are any errors, you can see it and find the problem (what lines of CSV has problem).