PHP Foreach大数据

Question

There is MySQL table with 24500 rows of data, and there is text file with 26000 string of data needed to be inserted into MySQL, problem is what this 26000 strings duplicates data in MySQL table, so we need to compare them, and insert only new/unique.

cadastreArray - array from text file

districtArray - mysql array

When i try to do

foreach ($cadastreArray as $cadastreValue) {
    $districtExist = false;
    foreach ($districtArray as $districtData) {
        if ($cadastreValue[0] == $districtData['1']) {
            $districtExist = true;
            break;
        }
    }
}

if(!$districtExist) { MySQL INSERT ... }

i am getting execution time error, and even 3 minutes are not enought. Maybe you can offer better/faster way?

Answer 1

May be you can set the mysql field as unique so when you will be going to insert it will not insert and will generate error number and will continue execution. So you need not to compare.

One more thing that you can do is you can increase max_execution_time in php.ini

Answer 2

Another option. Load your 26k text file into a temp table (LOAD DATA INFILE... will do this quickly).

Then you can do an insert based on a query that takes your temp table and LEFT JOINs that against your full table, checking that a field on the full table is NULL.

Simple example script here:-

<?php

$file = "SomeTextFile.txt";

$sql = "CREATE TEMPORARY TABLE cadastre
(
    field1 INT,
    field2 VARCHAR(255),
    etc...
)";

if(!($db->query($sql)))
{
    die($db->error());// if error, stop script
}

if(!($db->query("LOAD DATA INFILE '$file' INTO TABLE cadastre")))
{
    die($db->error());// if error, stop script
}

$sql = "INSERT INTO district (field1, field2, field3, ......)
        SELECT a.field1, a.field2, a.field3
        FROM cadastre a
        LEFT OUTER JOIN district b
        ON a.field1 = b.field1
        WHERE b.field1 IS NULL";

if(!($db->query($sql)))
{
    die($db->error());// if error, stop script
}

?>

Make sure the temp table and the table you are inserting into have useful indexes added.