Symfony3 / Doctrine 2 flush error

Question

I created a symfony command that should import 2M entries and insert them in the database.

I put a flush/clear and yield every 500 entries to make it possible to insert it with doctrine and track the progression of the insertion. On my machine with a mySQL database server it works, but on the distant server with Postgresql it doesn't work, i get a notice :

 [Symfony\Component\Debug\Exception\ContextErrorException]  
  Notice: Undefined index: 0000000025969b1e00000000029e7855  

Exception trace:
 () at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:2917
 Doctrine\ORM\UnitOfWork->getEntityIdentifier() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php:656
 Doctrine\ORM\Persisters\Entity\BasicEntityPersister->prepareUpdateData() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php:692
 Doctrine\ORM\Persisters\Entity\BasicEntityPersister->prepareInsertData() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php:271
 Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:1014
 Doctrine\ORM\UnitOfWork->executeInserts() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:378
 Doctrine\ORM\UnitOfWork->commit() at /var/nginx/kweeri-next/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:356
 Doctrine\ORM\EntityManager->flush() at /var/nginx/kweeri-next/var/cache/dev/appDevDebugProjectContainer.php:6083
 DoctrineORMEntityManager_00000000612352fa000000006464aa610448bf5c446411a05ad87329e18cead6->flush() at /var/nginx/kweeri-next/src/AppBundle/Utility/Clients/IRI/PPS/ImportItems.php:219

This is the code where the flush puts a notice.

 public function importDictionnary($companyMachineName, $file) {
        $company = $this->em->getRepository("AGAdminBundle:Company")->findOneBy(["machineName" => $companyMachineName]);
        if (!$company) {
            throw new EntityNotFoundException("no company found for $companyMachineName");
        }
        $eans = $file;
        $this->logger->addInfo("File downloaded");

        $product_repo = $this->em->getRepository(\AppBundle\Entity\PPS\Product::class);
        $existings = array_map(function (\AppBundle\Entity\PPS\Product $product) {
            return $product->getEan();
        }, $product_repo->findBy(["company" => $company]));
        $old_value = -1;
        foreach ($eans as $k => $ean) {
            if (!in_array($ean, $existings)) {
                $this->em->persist(new \AppBundle\Entity\PPS\Product($ean, $company));

                if ($k % 500 == 0) {
                    yield round(($k / count($eans)) * 100);
                    $this->em->flush();
                    $this->em->clear();
                }
            }
        }
        $this->em->flush();
    }

EDIT: it takes arround 30 seconds to yield 2 on my machine. On the distant machine, it takes up to 20 minutes before it yields 2 and stops.

Answer 1

When you call $em->clear() the whole identity map is cleared. You need to reload all objects from database to be able to work with them, otherwise doctrine could not find the object in identity map which results in undefined index error.

In case you know the object id and you don't need to work with object properties, you can use a reference instead of loading the object from database.