批量导入文本文件导致Doctrine和Symfony（PHP）出现内存问题

Question

I'm importing a textfile of 15MB into a mysql-database. When I'm doing the following, the data are not imported but the memory usage is still constant about 28MB.

$handle = fopen("textfile.txt","r");
while (($data = fgetscsv($handle, 1024, "|")) !== false) {
    // processing data
}
fclose($handle);

When I bring now Doctrine into play, the memory usage still grows up and up until memory is overflow and the script crashes.

gc_enable(); // Enable Garbage Collector
$handle = fopen("textfile.txt","r");
$i=0;
while (($data = fgetscsv($handle, 1024, "|")) !== false) {
    $myEntity = $this->doctrine->getRepository('MyBundle:MyEntity')->find($data[0]);
    if (!$myEntity) {
        $myEntity = new MyEntity();
        $myEntity->setId($data[0]);
        $myEntity->setName($data[1]);
    } else {
        $myEntity->setName($data[1]);
    }
    $this->em->persist($myEntity);
    if ($i%100==0) {
       $this->em->flush();
       $this->em->clear();
       gc_collect_cycles();
    }
    $i++;
}
fclose($handle);
$this->em->flush();
gc_disable(); // Disable Garbage Collector

The memory usage grows up to 256 MB and then the script causes memory issue because of the limit which is set to 256 MB. So what (else) can I do to keep the memory usage low?

Answer 1

Try delaying every Doctrine's call as much as you can. In this particular code, Doctrine 's find() gets called for each file row. You could optimize this.

$dataChunk = array();
while (($data = fgetscsv($handle, 1024, "|")) !== false) {

    # collect the data into array
    $dataChunk[$data[0]] = array( $data[0], $data[1] );

    if ($i > 0 && $i%100==0) {
        # lets prepare and persist the data
        $idsToRead = array_keys($dataChunk);

        # read all entitties with given IDs
        $entities = $this->doctrine->getRepository('MyBundle:MyEntity')->findEntitiesById($idsToRead);
        $existentIds = array_map($entities, function($e){ return $e->getId(); });

        # find out which Ids are non-existent
        $newIds = array_diff($idsToRead, $existentIds);

        # Overwrite the name for existent entity
        foreach ($entities as $e){
            $e->setName($dataChunk[$e->getId()][1]);
        }

        # Create new entities for non-existent IDs
        foreach ($newIds as $id){
            $e = new MyEntity();
            $e->setId($id);
            $e->setName($dataChunk[$id][1]);
            $this->em->persist($e);
        }

        # finally, flush the data
        $dataChunk = array();
        $this->em->flush();
        $this->em->clear();
        gc_collect_cycles();            
    }

    $i++;
}

Some things to pay attention to:

• findEntitiesById would be custom repository method. It does not exist out-of-box
• database communication get executed on every 100th record
• be sure to skip 0th row (i = 0, i % 100 == 0 => TRUE ). I have added the condition
• be sure to apply the same logic if you do not reach modulus 100 at the end (number of rows is not dividable by 100)