PHP Crawler从服务器获取所有内存

Question

I wrote a really simple PHP crawler, but I have problem with the memory loss. The code is:

<?php
require_once 'db.php';

$homepage = 'https://example.com';
$query = "SELECT * FROM `crawled_urls`";
$response = @mysqli_query($dbc, $query);

$already_crawled = [];
$crawling = [];

while($row = mysqli_fetch_array($response)){
  $already_crawled[] = $row['crawled_url'];
  $crawling[] = $row['crawled_url'];
}

function follow_links($url){
  global $already_crawled;
  global $crawling;
  global $dbc;

  $doc = new DOMDocument();
  $doc->loadHTML(file_get_contents($url));

  $linklist = $doc->getElementsByTagName('a');

  foreach ($linklist as $link) {
    $l = $link->getAttribute("href");
    $full_link = 'https://example.com'.$l;

    if (!in_array($full_link, $already_crawled)) {

      // TODO: Fetch data from the crawled url and store it in the DB. Check if it was already stored.

      $query = 'INSERT INTO `crawled_urls`(`id`, `crawled_url`) VALUES (NULL,\'' . $full_link . '\')';
      $stmt = mysqli_prepare($dbc, $query);
      mysqli_stmt_execute($stmt);

      echo $full_link.PHP_EOL;
    }
  }

  array_shift($crawling);

  foreach ($crawling as $link) {
    follow_links($link);
  }
}

follow_links($homepage);

Can you help me out and share with me a way to avoid this huge memory loss? When I start the process it is all working fine, but the memory is steadily rising up to 100%.

Answer 1

You need to unset $doc when you no longer need it:

function follow_links($url){
  global $already_crawled;
  global $crawling;
  global $dbc;

  $doc = new DOMDocument();
  $doc->loadHTML(file_get_contents($url));

  $linklist = $doc->getElementsByTagName('a');

  unset($doc);

  foreach ($linklist as $link) {
    $l = $link->getAttribute("href");
    $full_link = 'https://example.com'.$l;

    if (!in_array($full_link, $already_crawled)) {

      // TODO: Fetch data from the crawled url and store it in the DB. Check if it was already stored.

      $query = 'INSERT INTO `crawled_urls`(`id`, `crawled_url`) VALUES (NULL,\'' . $full_link . '\')';
      $stmt = mysqli_prepare($dbc, $query);
      mysqli_stmt_execute($stmt);

      echo $full_link.PHP_EOL;
    }
  }

  array_shift($crawling);

  foreach ($crawling as $link) {
    follow_links($link);
  }
}

follow_links($homepage);

Explanation: You are using recursion, that is, you are using a stack of functions basically. This means that if you have a stack of 20 elements, all the resources for all the functions in your stack will be allocated accordingly. The deeper this gets the more memory you use. $doc is the main problem, but you may want to look at the usage of your other variables and make sure nothing unneeded is allocated when you call the function again.

Answer 2

Try to unset the $doc variable before calling the function:

function follow_links($url){
  global $already_crawled;
  global $crawling;
  global $dbc;

  $doc = new DOMDocument();
  $doc->loadHTML(file_get_contents($url));

  $linklist = $doc->getElementsByTagName('a');

  foreach ($linklist as $link) {
    $l = $link->getAttribute("href");
    $full_link = 'https://example.com'.$l;

    if (!in_array($full_link, $already_crawled)) {

      // TODO: Fetch data from the crawled url and store it in the DB. Check if it was already stored.

      $query = 'INSERT INTO `crawled_urls`(`id`, `crawled_url`) VALUES (NULL,\'' . $full_link . '\')';
      $stmt = mysqli_prepare($dbc, $query);
      mysqli_stmt_execute($stmt);

      echo $full_link.PHP_EOL;
    }
  }

  array_shift($crawling);
  unset($doc);

  foreach ($crawling as $link) {
    follow_links($link);
  }
}

Answer 3

The main problem of your code is that you are using recursion. That way, you are keeping old pages in memory, although you don't need them any more.

Try removing that recursion. It should be relatively easy, since you are already using lists to store your links. I would prefer to use one list and represent URLs as objects, however.

Some other things:

• Looks like you have a SQL injection vulnerability, so learn to use prepared statements correctly
• Avoid using global variables (you can make your function return a list of links)
• If you plan to use this code on other people's websites ensure you obey robots.txt , limit your crawl rate and make sure you don't crawl pages multiple times

If you want to use this code for something other than education, I suggest you use a library. This will be easier than creating a crawler from scratch.