PHP Multiple cURL requests to REST API stalls

Question

Currently I have a system that sends multiple requests to a REST API. It is structured something like this:

foreach ($data as $d)
{
    $ch = curl_init( $url );
    curl_setopt( $ch, CURLOPT_POST, 1);
    curl_setopt( $ch, CURLOPT_HTTPHEADER, (array of data here));
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
    $response = curl_exec( $ch );
    $retry = 0;
    while((curl_errno($ch) == 7 || curl_errno($ch) == 52) && $retry < 3)
    {
        $response = curl_exec($ch);
        $retry++;
    }
    curl_close($ch);
    (decode XML Response and loop)
}

(I can't expose the whole code so I have filled in the operations that are happening in brackes)

However after a few hundred requests the FastCGI script stalls. The REST API will still respond during this period if I query it in another fashion, but this batch client will not send anymore requests. After a few minutes, it will start responding again. I'm not sure why this is stalling, I can see via htop that there is no CPU activity on the threads at either end whilst this is happening.

Is there any reason why the cURL/PHP script would stall here?

Answer 1

if you allowed to use external PHP libraries; I'd like to suggest this method: https://github.com/php-curl-class/php-curl-class

    // Requests in parallel with callback functions.
$multi_curl = new MultiCurl();

$multi_curl->success(function($instance) {
    echo 'call to "' . $instance->url . '" was successful.' . "\n";
    echo 'response: ' . $instance->response . "\n";
});
$multi_curl->error(function($instance) {
    echo 'call to "' . $instance->url . '" was unsuccessful.' . "\n";
    echo 'error code: ' . $instance->error_code . "\n";
    echo 'error message: ' . $instance->error_message . "\n";
});
$multi_curl->complete(function($instance) {
    echo 'call completed' . "\n";
});

$multi_curl->addGet('https://www.google.com/search', array(
    'q' => 'hello world',
));
$multi_curl->addGet('https://duckduckgo.com/', array(
    'q' => 'hello world',
));
$multi_curl->addGet('https://www.bing.com/search', array(
    'q' => 'hello world',
));

$multi_curl->start();