PHP中带有pthreads的多线程文件处理

Question

I'm trying to create a script that process a number of files simultanously, the rule is, each file can only be processed once, and the input file is deleted after it has been processed. I created this script :

<?php

// Libraries for reading files
require_once "spooler.php";

// Configuration section ///////////////////////////////////////////////////////

$config["data"] = "data";
$config["threads"] = 20;
$config["timer"] = 1;

// Array to store currently processed files
$config["processed_files"] = array();

// Processing section //////////////////////////////////////////////////////////

$timer = 0;
$pool = new Pool($config["threads"], \ProcessingWorker::class);

while (true) {

    // Read a number of files from the data folder according to the number of thread
    $files = Spooler::read_spool_file($config["data"], $config["threads"]);
    foreach ($files as $file) {
        // Check if the file is already processed
        if (in_array($file, $config["processed_files"])) continue;
        // Submit the file to the worker
        echo "Submitting $file\n";
        $config["processed_files"][$file] = $file;
        $pool->submit(new ProcessingJob($config, $file));
    }

    sleep($config["timer"]);
    $timer++;
}

$pool->shutdown();

// Processing thread section ///////////////////////////////////////////////////

class ProcessingJob extends Stackable {

    private $config;
    private $file;

    public function __construct($config, $file)
    {
        $this->config = $config;
        $this->file = $file;
        $this->complete = false;
    }

    public function run()
    {
        echo "Processing $this->file\n";
        // Pretend we're doing something that takes time
        sleep(mt_rand(1, 10));
        file_put_contents("_LOG", $this->file."\n", FILE_APPEND);

        // Delete the file
        @unlink($this->file);
        // Remove the file from the currently processing list
        unset($this->config["processed_files"][$this->file]);
    }

}

class ProcessingWorker extends Worker {
    public function run() {}
}

However, this code doesn't work well, it doesn't process the same files twice, but instead sometimes it skip processing some files. Here's the file list it should be processed, but it only process these files .

Where am I doing it wrong?

Answer 1

Output to the log file isn't synchronized, it's highly likely that two threads are concurrently calling file_put_contents on the log file and so corrupting it's output.

You should not write to a log file in this way.

If $config['processed_files'] is intended to be manipulated by multiple contexts then it should be a thread safe structure descended from pthreads, not a plain PHP array.