如何为大型CSV文件数据提取优化循环

Question

I have a question about code optimization. I haven't coded anything besides simple loops in over ten years.

I created the code below, which works fine but is super slow for my needs.

In essence, I have 2 CSV files:

• a source CSV file that has about 500 000 records, let's say: att1, att2, source_id, att3, att4 (in reality there are about 40 columns)
• a main CSV file that has about 120 million records, let's say: att1, att2, att3, main_id, att4 (in reality there are about 120 columns)

For each source_id in the source file, my code parses the main file for all the lines where main_ id == source_id and it writes each of those lines in a new file.

Do you have any suggestion on how I could optimize the code, to go much much faster?

<?php

$mf = "main.csv";
$mf_max_line_length = "512";
$mf_id = "main_id";

$sf = "source.csv";
$sf_max_line_length = "884167";
$sf_id = "source_id";


if (($mf_handle = fopen($mf, "r")) !== FALSE)
{
    // Read the first line of the main CSV file
    // and look for the position of main_id
    $mf_data = fgetcsv($mf_handle, $mf_max_line_length, ",");
    $mf_id_pos = array_search ($mf_id, $mf_data);

    // Create a new main CSV file
    if (($nmf_handle = fopen("new_main.csv", "x")) !== FALSE)
    {
        fputcsv($nmf_handle,$mf_data);
    } else {
        echo "Cannot create file: new_main.csv" . $sf;
        break;
    }
}

// Open the source CSV file
if (($sf_handle = fopen($sf, "r")) !== FALSE)
{
    // Read the first line of the source CSV file
    // and look for the position of source_id
    $sf_data = fgetcsv($sf_handle, $sf_max_line_length, ",");
    $sf_id_pos = array_search ($sf_id, $sf_data);

    // Go trhough the whole source CSV file
    while (($sf_data = fgetcsv($sf_handle, $sf_max_line_length, ",")) !== FALSE)
    {
        // Open the main CSV file
        if (($mf_handle = fopen($mf, "r")) !== FALSE)
        {
            // Go trhough the whole main CSV file
            while (($mf_data = fgetcsv($mf_handle, $mf_max_line_length, ",")) !== FALSE)
            {
                // If the source_id matches the main_id
                // then we write it into the new_main CSV file
                if ($mf_data[$mf_id_pos] == $sf_data[$sf_id_pos])
                {
                    fputcsv($nmf_handle,$mf_data);
                }
            }
            fclose($mf_handle);
        }
    }
    fclose($sf_handle);
    fclose($nmf_handle);
}

?>

Answer 1

Sounds like a job for mysql.

First, you'll need to create tables based on all your fields. See here

Then, you'll load your data. See here

Finally, you'll create a query like:

SELECT * INTO OUTFILE '/tmp/something.csv' 
    FIELDS TERMINATED BY ',' ENCLOSED BY '"' 
    LINES TERMINATED BY '\n' 
FROM source_table INNER JOIN main_table ON 
    source_table.source_id=main_table.main_id;