PHP内存耗尽，继承的代码会导致较大的文件出现错误，是否刷新内存，批处理或增加内存分配？

Question

Uploader worked fine until the file became larger than 100,000 lines. I didn't write the code but I want to fix it. I have worked with other languages but not PHP. I know there are different ways to address the issue, but I am unsure of the best investment of time. Ideally I would like uploader to accept files of any size. Changing the memory allocation seems to be the quickest fix, but I would expect long term issues when the file outgrows the memory. Flushing the memory and batching the uploads seem to be 2 sides of the same coin, however the uploader currently will only process a single file and a single upload to the database, every time the file is uploaded it deletes the previous data and replaces it with data from the file. Specifically I have been adjusting the CSV uploader and not the XLSX uploader.

I have already unsuccessfully tried to allocate addition memory to the program but it crashed the server and I would prefer not to do that again. I have also attempted to batch the csv file but it failed as well.

<?php 
class Part {
            public $id;
            public $oem;
            public $part_number;
            public $desc;

            // Assigning the values
            public function __construct($id, $oem, $part_number, $desc) {
                $this->id = $id;
                $this->oem = $oem;
                $this->part_number = $part_number;
                $this->desc = $desc;
            }
}
//imports single csv file and returns an array of Parts
function importCSVpartfinder($filename, $brand, $root){ //$filename is a dataTable of dimensions: first row contains dimension labels, second row are units, the first column is the part number
    $handle = fopen($filename, 'r') or die('unable to open file: $filename');
    $contents = fread($handle, filesize($filename));
    fclose($handle);
    $row = explode("\r" , $contents);
    $data = array();
    $data2 = array();
    for ($i=0; $i < sizeof($row); $i++) { 
        $columns = explode(",", $row[$i]);
        array_push($data, $columns);
        }
    $all = array(); //array of all Parts

//I should probably sanatize here

    for ($i=0; $i < sizeof($data); $i++) { 
        if (sizeof($data[$i]) != 1){
            $id = $data[$i][0];
            $oem = $data[$i][1];
            $part_number = $data[$i][2];
            $desc = $data[$i][3];
            $obj = new Part($id, $oem, $part_number, $desc);
            array_push($all, $obj);
        }
    }
    return $all;
}

//returns a message with # of succes and list of failures  //this is slow with large uploads
function addPartsToDB($data, $connection){      //$data is an array of Parts
    //delete
    $deleteSQL = "DELETE FROM Part_finder WHERE 1";
    $res = $connection->query($deleteSQL);
    if (!$res){
        echo " Failed to delete Part_finder data, ";
        exit;
    }
    //insert
    $e=0;
    $s=0;
    $failures = "";
    $d="";
    for ($i=0; $i < sizeof($data); $i++) { 
        $d .= "(".$data[$i]->id.",'".$data[$i]->oem."','".$data[$i]->part_number."','".$data[$i]->desc."'),";
        $s++;

    }
    $d = substr($d, 0, -1);
    $sqlquery = "INSERT INTO Part_finder (id_part, oem, part_number, description) VALUES $d";
    $res = $connection->query($sqlquery);
    if (!$res){
        $sqlError = $connection->error;
        return ( $s." items failed to update. Database error. ".$sqlError);
    }else{
        return ( $s." items updated."); 
    }

/*
    for ($i=0; $i < sizeof($data); $i++) { 
        $d = "(".$data[$i]->id.",'".$data[$i]->oem."','".$data[$i]->part_number."','".$data[$i]->desc."')";
        $sqlquery = "INSERT INTO Part_finder (id_part, oem, part_number, description) VALUES $d";
        #$res = $connection->query($sqlquery);
        if (!$res){
            $failures .= $data[$i]->part_number . "
" ;
            $e++;
        }else{
            $s++;   
        }
    }*/
    #return $sqlquery;

}

function importXLSXpartfinder($filename, $root){
    require($root.'./plugins/XLSXReader/XLSXReader.php');
    $xlsx = new XLSXReader($filename);
/*  $sheetNames = $xlsx->getSheetNames();
    foreach ($sheetNames as $Name) {
        $sheetName = $Name;
    }*/
    $sheet = $xlsx->getSheet("Sheet1");
    $rawData = $sheet->getData();
    #$columnTitles = array_shift($rawData);
    $all = array(); //array of all Parts
    for ($i=0; $i < sizeof($rawData); $i++) { 
        if (sizeof($rawData[$i]) != 1){
            $id = $rawData[$i][0];
            $oem = $rawData[$i][1];
            $part_number = $rawData[$i][2];
            $desc = $rawData[$i][3];
            $obj = new Part($id, $oem, $part_number, $desc);
            array_push($all, $obj);
        }
    }
    return $all;
}

$filename = $file["partfinder"]["tmp_name"];
if($file["partfinder"]["size"] > 100000000){
    echo "File too big".$file["partfinder"]["size"];
    exit;
}
//$file comes from edit.php
if($file["partfinder"]["type"] === "text/csv"   ) {
    $a = importCSVpartfinder($filename, $brand, $root);
}elseif ($file["partfinder"]["type"] === "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" ) {
    $a = importXLSXpartfinder($filename, $root);
}else{
    var_dump($file["partfinder"]["type"]);
    echo ".xlsx or .csv file types only";
    exit;   
}
$b = addPartsToDB($a,$connection);
echo $b;

?>

The memory exhaustion currently occurs on line 25

$columns = explode(",", $row[$i]);

and the error code is

Fatal error: Allowed memory size of 94371840 bytes exhausted (tried to allocate 20480 bytes) in /www/tools/import-csv-partfinder.php on line 25

Ideally I would still like to upload a single file to update the database and I would need to alter additional programs to be able to upload multiple files or not have the database wipe itself during every upload. Unfortunately I am not able to contact the person who wrote the programs originally, so I am pretty much on my own to figure this out.

Answer 1

I'd suggest using a generator to read your CSV rather than reading the whole thing into an array (actually two arrays with the way it's currently written). This way you only hold one line at a time in memory.

function importCSVpartfinder($filename = '') {
    $handle = fopen($filename, 'r');
    while (($row = fgetcsv($handle)) !== false) {
        yield $row;
    }
    fclose($handle);
}

Then for your database insert function, use a prepared statement and iterate the generator, executing the statement for each row in the file.

function addPartsToDB($parts, $connection) {
    $connection->query('DELETE FROM Part_finder');
    $statement = $connection->prepare('INSERT INTO Part_finder
                                       (id_part, oem, part_number, description)
                                       VALUES (?, ?, ?, ?)');
    foreach ($parts as $part) {
        $statement->execute($part);
    }
}

These examples are simplified just to show the concept. You should be able to adapt them to your exact needs, but they are working examples as written.

addPartsToDB(importCSVpartfinder($filename), $connection);