我的并发编程逻辑怎么了？

Question

I wrote a web spider to spider pages concurrently. For each link that the spider finds, I want to fork off a new child that starts the process all over again.

I don't want to overload the target server so I created a static array that all objects can access. Each child can add their PID to the array, and either parent or child should check the array to see if $maxChildren have been met, and if so, patiently wait until any child finishes.

As you see, I have $maxChildren set to 3. I am expecting to see 3 simultaneous processes at any given time. However, that's not the case. The linux top command shows 12 to 30 processes at any given time. In concurrent programming, how can I regulate the number of simultaneous processes? My logic is currently inspired by how Apache handles it's max children, but I'm not exactly sure how that works.

As pointed out in one of the answers, globally accessing the static variable brings up issues with race conditions. To deal with this, the $children array takes the unique $PID of the process as both the key and it's value, thereby creating a unique value. My thinking is that since any object can only deal with one $children[$pid] value, locking is not necessary. Is this not true? Is there a chance that two processes could try to unset or add the same value at some point?

private static $children = array();

private $maxChildren = 3;

public function concurrentSpider($url) {

        // STEP 1:
        // Download the $url
        $pageData = http_get($url, $ref = '');

        if (!$this->checkIfSaved($url)) {
            $this->save_link_to_db($url, $pageData);
        }

        // STEP 2:
        // extract all hyperlinks from this url's page data
        $linksOnThisPage = $this->harvest_links($url, $pageData);

        // STEP 3:
        // Check the links array from STEP 2 to see if this page has
        // already been saved or is excluded because of any other
        // logic from the excluded_link() function
        $filteredLinks = $this->filterLinks($linksOnThisPage);

        shuffle($filteredLinks);

        // STEP 4: loop through each of the links and
        // repeat the process
        foreach ($filteredLinks as $filteredLink) {

            $pid = pcntl_fork();
            switch ($pid) {
                case -1:
                    print "Could not fork!\n";
                    exit(1);
                case 0:
                    if ($this->checkIfSaved($filteredLink)) {
                        exit();
                    }
                    //$pid = getmypid();
                    print "In child with PID: " . getmypid() . " processing $filteredLink \n";


                    $var[$pid]->concurrentSpider($filteredLink);
                    sleep(2);

                    exit(1);
                default:
                    // Add an element to the children array
                    self::$children[$pid] = $pid;
                    // If the maximum number of children has been
                    // achieved, wait until one or more return
                    // before continuing.

                    while (count(self::$children) >= $this->maxChildren) {
                        //print count(self::$children) . " children \n";
                        $pid = pcntl_waitpid(-1, $status);
                        unset(self::$children[$pid]);
                    }
            }
        }
    }

This is written in PHP. I know that the pcntl_waitpid function with argument of -1 waits for any child to complete regardless of the parent ( http://php.net/manual/en/function.pcntl-waitpid.php ).

What's wrong with my logic and how can I correct it so that only $maxChildren processes are running simultaneously? I'm also open to improving the logic in general if you have suggestions.

Answer 1

First thing to note: if this is truly a global being shared among multiple threads, it's possible that multiple threads are adding to it at once and you're running afoul of a race condition. You need some sort of concurrency control to ensure that only one process is accessing your global array at once.

Also, try the simple debugging trick of having each process write out (to the console or to a file) its PID and the full contents of the global array each time a new spider is forked. It will help you to check your assumptions (which are plainly wrong at some point) and figure out what's going wrong.

EDIT: (In response to the comments)

I'm not a PHP developer, but if I had to guess, based on the fact that you're using an OS tool that counts OS-level processes, I'd guess that your fork is spawning multiple processes, but your static array is global within the current process . Implementing system-wide shared memory is a lot more complicated!

If you just want to count something and ensure that instances of a shared resource don't grow out of control, look into semaphores , and see if you can find a way in PHP to create a named semaphore object that can be shared between multiple instances of your spider.

Answer 2

Use a real programming language ;)

Step 1 is kind of bad why are you downloading if it might be in the db. Put that inside the if and see if you can put a mutex around it. Maybe so something in sql to imitate one.

I hope harvest_links uses a proper html processor with css selector support (i like fizzler for .NET). I guess regular expression would be fine if its just to get links but it is possible to mess up.

I see step 4 and i don't think its bad but personally i'd do it a different way. I'd have something like step one to insert url,page,flag into a db. Then i'd have another process or the same one ask the db for unprocessed pages and set the flag to some value if it errors and another if its successful. This is so if something fails of the process exits (shutdown, crash, power out, etc) it can pick it up easily and don't need to scan every page to find where it left off. It just ask the database for the next link and redoes what it didnt finish

Answer 3

PHP doesn't support multithreading, therefore it doesn't support mutexes or any other synchronization methods. As others have said in their answers, this will lead to a race condition.

You'll have to write a wrapper in C or bash. That way, the PHP script can submit targets to the wrapper, and the wrapper will handle scheduling.

Another approach is to rewrite your spider in Python or Ruby, both of which support multithreading. That will eliminate the need for interprocess communication.

Edit: On second thought, the best way is to write the wrapper in Python or Ruby and reuse your existing PHP code as a black box. That's a compromise of the solutions above.

Answer 4

If the spider is for practical purposes, you might want to google "curl multithread"

cURL Multi Threading with PHP