Connection errors in forked PHP processes

Question

I have a PHP script that takes N documents from MongoDB, forks the process into K child PHP processes, each process does some things with each document and tries to update document's info (see the code below).

On my local environment ( Docker ) everything is cool, but on the server (no Docker there) sometimes during the loop strange things happen...

Randomly all forked processes can not connect to MongoDB. The updateOne command returns an error :

"Failed to send "update" command with database "databasename": Invalid reply from server. in /vendor/mongodb/mongodb/src/Operation/Update.php on line 158".

This happens to all processes at the same time only for one (or several) random loop iterations. When each process goes to another iteration (takes the next document) -- everything is ok again. I make 5 tries to write to MongoDB.

Each try is with delay +1 sec to the previous, so the first try makes immediately, if any exception is caught -- wait a second and try again, the next try will be in 2 seconds and so on. But this does not help, all these 5 tries are broken.

This is not mongoDB problem, it's log is empty and it even don't receive anything from PHP, when error happens.

Also I have admitted, the more simultaneous processes I run -- the more frequent errors occur.

Also it is not server resource problem, when error occurs, half of RAM (4 gig) is free and CPU is working for the half of it's power.

Maybe PHP has some configuration for this? Some memory limits or something...

I use PHP v 7.1.30 MongoDB v 3.2.16 PHP Package mongodb/mongodb v 1.1.2

<?php

$processesAmount = 5;
$documents = $this->mongoResource->getDocuments();

for ($processNumber = 0; $processNumber < $processesAmount; $processNumber++) {
    // create child process
    $pid = pcntl_fork();

    // do not create new processes in child processes
    if ($pid === 0) {
        break;
    }

    if ($pid === -1) {
       // some errors catching staff here...
    }
    else if ($pid === 0) {
        // create new MongoDB connection
    }
    else { 
        // Protect against Zombie children
        // main process waits before all child processes end
        for ($i = 0; $i < $processesAmount; $i++) {
            pcntl_wait($status);
        }
        return null;
    }

    // spread documents to each process without duplicates
    for ($i = $processNumber; $i < count($documents); $i += $processesAmount) {
        $newDocumentData = $this->doSomeStaffWithDocument($documents[$i]);
        $this->mongoResource->updateDocument($documents[$i], $newDocumentData);
    }
}

Answer 1

There could be many issues here, one being that all processes are sharing 1 DB connection and the first to connect is then disconnecting and killing the connection for them all. Check the second example in the docs here: https://www.php.net/manual/en/ref.pcntl.php

If that doesn't help, the way I read your code the "spreading" part is happening in every process, when it should be happening once. Shouldn't you be putting the "work" in the child section like below?

$processesAmount = 5;
$documents = $this->mongoResource->getDocuments();
$numDocs = count($documents);
$i = 0;
$children = [];

    for ($processNumber = 0; $processNumber < $processesAmount; $processNumber++) {
        // create child
        $pid = pcntl_fork();

        if ($pid === -1) {
           // some errors catching staff here...
        } else if ($pid) {
            //parent
            $children[] = $pid;
        } else { 
            //child
           while (!empty($documents) && $i <= $numDocs) {
               $i += $processNumber;
               $doc = $documents[$i] ?? null;
               unset($documents[$i]);
               $newDocumentData = $this->doSomeStaffWithDocument($doc);
               $this->mongoResource->updateDocument($doc, $newDocumentData);
           }
        }
    }

    //protect against zombies and wait for parent
    //children is always empty unless in parent
    while (!empty($children)) {  

        foreach ($children as $key => $pid) {

            $status = null;

            $res = pcntl_waitpid($pid, $status, WNOHANG);

            if ($res == -1 || $res > 0) {   //if the process has already exited
                unset($children[$key]);
            }

        }

    }

}