如何加快CURL任务？

Question

I'm using CURL to fetch some data from user accounts. First it logs in and then redirect to another URL where the data resides.

My stats showed that it took an average of 14 seconds to fetch some data spread over 5 pages. I would like to speed things up, my questions are:

Is it possible to see how much each step takes? Do you know how could I speed up/enhance CURL?

Thanks

Answer 1

To make the task 'feel' faster, don't run it as part of a web request, run it in the background as a periodic task (cron job).

Cache the response on disk or in a database.

Answer 2

you can use parallelCurl by Pete Warden. The source is available here http://github.com/petewarden/ParallelCurl . The module allows you to run multiple CURL url fetches in parallel in PHP

Answer 3

You can't make the process of retrieving a page from a server any faster.

You can make the pages smaller, so they can download quicker. You can beef up the processing power on the servers or the connection between your server and the server the pages are on.

If you are consuming a service, what format is the data in? If it is XML, maybe it is too verbose and this is causing lots of extra kilobytes, for example.

Answer 4

Speed up curl with this opcion

curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );

Regards

Answer 5

split task to 3 files.

1. file for retrieving page list and as your main script (to put on crontab) (main.php)
2. for parsing the actual page. (parse.php)
3. some shell script to process your 2nd script.

Then, in your 1st file, do something like this:

<?php
$pagelist = get_page_list();//this will retrieve page list using CURL and save each page to some, let's say pagelist.txt and return this absolute path.

exec("/path/to/php /your/3rdscript.sh < $pagelist");
?>

And here's your 3rd file:

#!/bin/bash  

while read line
do
    /path/to/php /path/to/your/2ndscript.php -f $line &
done

Please note that on 3rd script (the shell script) I use & (ampersand). This will tell the shell to put that particular process into background process.

On your 2nd script, you can use something like this:

<?php

$pageurl = $argv[2];
//do your curl process to fetch page $pageurl here

Using step above, you can speed up by fetching several pages at once.