PHP cUrl loop leaking memory

Question

The following code is in a loop. Each loop changes URL to a new address. My problem is that each pass takes up more and more memory.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://site.ru/');
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, 'http://site.ru/');
curl_setopt($ch, CURLOPT_HEADER, false);

$html = new \DOMDocument();
$html->loadHTML(curl_exec($ch));

curl_close($ch);
$ch = null;

$xpath = new \DOMXPath($html);
$html = null;

foreach ($xpath->query('//*[@id="tree"]/li[position() > 5]') as $category) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $xpath->query('./a', $category)->item(0)->nodeValue);
    curl_setopt($ch, CURLOPT_TIMEOUT, 60);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_AUTOREFERER, 'http://site.ru/');
    curl_setopt($ch, CURLOPT_HEADER, false);

    $html = new \DOMDocument();
    $html->loadHTML(curl_exec($ch));

    curl_close($ch);
    $ch = null;

    // etc.
}

The memory is 2000 Mb. Script execution time ~ 2h. PHP version 5.4.4. How to avoid memory leak? Thanks!

Answer 1

Stories from the internet indicate that curl_setopt($ch, CURLOPT_RETURNTRANSFER, true) is broken in for some PHP/cURL versions:

• #13225 Memmory leak in curl
• #40831 cURL extension doesnt clear buffer for reused handle
• PHP script memory leak issue
• PHP cURL, memory leak when using CURLOPT_RETURNTRANSFER

You can also find stories for DOM :

• DOMDocument PHP Memory Leak
• PHP/DOMDocument: unset() does not release resources
• DOMDocument / Xpath leaking memory during long command line process - any way to deconstruct this class

Create a minimal test case which spots the cause of the leak. Ie remove the unrelated package (DOM or cURL) from the code.

Then reproduce it with the latest PHP version. If it's still causing the leak, file a bug report else use that PHP version.

Answer 2

Reuse the same curl handle instead of creating and destroying it each time in your loop.

$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, 'http://site.ru/');
curl_setopt($ch, CURLOPT_HEADER, false);
foreach ($pages as $url) {
    curl_setopt($ch, CURLOPT_URL, $url);
    $response = curl_exec($ch);
}
curl_close($ch);