CURL slower than fread while streaming file? how to speed up CURL

Question

I'm trying to make a script to download a part of file. Just make a test with CURL and fread, i realize that CURL during streaming process is slower than fread. Why? how to speed up curl for stream a file? i don't like to use fread , fopen because i need limited time during streaming process.

Here is my sample code.

$start = microtime(true);
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
$response = fread($f, 3); echo $response.'<br>';
$response = fread($f, 3); echo $response.'<br>';
$response = fread($f, 3); echo $response.'<br>';
$response = fread($f, 3); echo $response.'<br>';
$response = fread($f, 3); echo $response.'<br>';

$stop = round(microtime(true) - $start, 5);
echo "{$stop}s";
exit();

fread / fopen It take only 1.1s

$start = microtime(true);
$curl = curl_init('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg');
curl_setopt($curl, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);

curl_setopt($curl, CURLOPT_RANGE, "0-2");
$response = curl_exec($curl);echo $response.'<br>';

curl_setopt($curl, CURLOPT_RANGE, "3-5");
$response = curl_exec($curl);echo $response.'<br>';

curl_setopt($curl, CURLOPT_RANGE, "6-8");
$response = curl_exec($curl);echo $response.'<br>';

curl_setopt($curl, CURLOPT_RANGE, "9-11");
$response = curl_exec($curl);echo $response.'<br>';

curl_setopt($curl, CURLOPT_RANGE, "12-14");
$response = curl_exec($curl);echo $response.'<br>';
curl_close($curl);

$stop = round(microtime(true) - $start, 5);
echo "{$stop}s";
exit();

curl took around 2.5s. if i take more step to download more a part of the file. curl will take more slower.

Why curl is slower? and what solution is it?

Answer 1

It is always slower because you added additional round-trip of HTTP calls. Every curl_exec is a single HTTP request.

Answer 2

your question is about having many partial requests or about incremental buffered reading for the same request.

The fopen/fread implementation fires a single HTTP request and read it piece by piece many times.

on the other hand the curl implementation fires many HTTP requests, one request per piece ( see partial range requests ). so we are comparing apples and oranges

to be fair the fopen/fread would look like this

$start = microtime(true);
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
$response = fread($f, 3); echo $response.'<br>';
fclose($f)
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
fseek($f, 3);
$response = fread($f, 3); echo $response.'<br>';
fclose($f)
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
fseek($f, 6);
$response = fread($f, 3); echo $response.'<br>';
fclose($f)
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
fseek($f, 9);
$response = fread($f, 3); echo $response.'<br>';
fclose($f)
$f = fopen('http://news.softpedia.com/images/news2/Debian-Turns-15-2.jpeg','r');
fseek($f, 12);
$response = fread($f, 3); echo $response.'<br>';
fclose($f)
$stop = round(microtime(true) - $start, 5);
echo "{$stop}s";
exit();

UPDATE: I've rewrite my answer

Answer 3

You can use curl_getinfo to see what is taking long.

It is possible the slowness could be due to the DNS lookup by the curl library. Have you tried using IP address instead of domain for the request url?

EDIT : alternatively, you can set CURLOPT_DNS_USE_GLOBAL_CACHE to true and set CURLOPT_DNS_CACHE_TIMEOUT to a longer value (eg 1 hour) - it's 2minutes by default.

Src: http://php.net/manual/en/function.curl-setopt.php

Answer 4

Try to set Keep-Alive header before first curl_exec , for example 300 seconds in such way:

$headers = array("Keep-Alive: 300");
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);

Answer 5

You can try the following solution:

curl_setopt($curl, CURLOPT_RANGE, "0-2, 3-5, 6-8, 9-11, 12-14");
$response = curl_exec($curl);echo $response.'<br>';
curl_close($curl);
$stop = round(microtime(true) - $start, 5);
echo "{$stop}s";
exit();