如何在PHP中使用curl获取所有页面分页数据

Question

I have implemented one data scraping script which fetch data using CURL.

But that fetch record only one page but i want all data because on that page pagination is there.

We are getting data using

$xpath = new DOMXPath($dom);
$xpath_resultset =  $xpath->query("//div[@class='parentListing']");

in above code parentListing is div class and i am fetching record between div and grep the data using different different class.

Can any one have idea how to get pagination data of all page.

If any one have any suggestion then please help me.

Thanks...

Answer 1

I think there is no generic pagination type. You should know pagination element div,ul,li or its name,class name etc. And then you can access pagination links, and sends requests to this links

$xpath = new DOMXPath($dom);
$xpath_resultset =  $xpath->query("//div[@class='parentListing']/a");

Otherwise you must walk all dom and decide which element can be paginate element. Some pages adds next and previous meta links. You can search these tags :

$xpath = new DOMXPath($dom);
$next_resultset =  $xpath->query("//link[@rel='next']");
$prev_resultset =  $xpath->query("//link[@rel='prev']");