[英]What PHP web crawler libraries are available?
I'm looking for some robust, well documented PHP web crawler scripts. 我正在寻找一些健壮的,记录良好的PHP Web爬虫脚本。 Perhaps a PHP port of the Java project - http://wiki.apache.org/nutch/NutchTutorial 也许是Java项目的PHP端口 - http://wiki.apache.org/nutch/NutchTutorial
I'm looking for both free and non free versions. 我正在寻找免费和非免费版本。
https://github.com/fabpot/Goutte也是一个兼容psr-0标准的好库。
There is a greate tutorial here which combines guzzlehttp and symfony/dom-crawler 这里有一个greate教程,它结合了guzzlehttp和symfony / dom-crawler
In case the link is lost here is the code you can make use. 如果链接丢失,这里是您可以使用的代码。
use Guzzle\Http\Client;
use Symfony\Component\DomCrawler\Crawler;
use RuntimeException;
// create http client instance
$client = new GuzzleHttp\ClientClient('http://download.cloud.com/releases');
// create a request
$response = $client->request('/3.0.6/api_3.0.6/TOC_Domain_Admin.html');
// get status code
$status = $response->getStatusCode();
// this is the response body from the requested page (usually html)
//$result = $response->getBody();
// crate crawler instance from body HTML code
$crawler = new Crawler($response->getBody(true));
// apply css selector filter
$filter = $crawler->filter('div.apismallbullet_box');
$result = array();
if (iterator_count($filter) > 1) {
// iterate over filter results
foreach ($filter as $i => $content) {
// create crawler instance for result
$cralwer = new Crawler($content);
// extract the values needed
$result[$i] = array(
'topic' => $crawler->filter('h5')->text();
'className' => trim(str_replace(' ', '', $result[$i]['topic'])) . 'Client'
);
}
} else {
throw new RuntimeException('Got empty result processing the dataset!');
}
You can use PHP Simple HTML DOM Parser . 您可以使用PHP Simple HTML DOM Parser 。 It's really simple and useful. 它非常简单实用。
if you are thinking about a strong base component than give a try to http://symfony.com/doc/2.0/components/dom_crawler.html 如果你正在考虑一个强大的基础组件而不是尝试http://symfony.com/doc/2.0/components/dom_crawler.html
it is amazing, having a features like css selector. 它很棒,有像css选择器这样的功能。
I know it is a bit old question. 我知道这是一个有点老问题。 A lot of useful libraries came out since then. 从那时起,出现了许多有用的库。
Give it a shot to Crawlzone . 给Crawlzone一个机会 。 It is fast, well documented, asynchronous internet crawling framework with a lot of great features: 它是一个快速,文档齐全的异步Internet爬网框架,具有许多强大的功能:
Also check out the article I wrote about it: 另请查看我写的关于它的文章:
https://www.codementor.io/zstate/this-is-how-i-crawl-n98s6myxm https://www.codementor.io/zstate/this-is-how-i-crawl-n98s6myxm
Nobody mentioned wget as a good starting point?. 没人提到wget是一个很好的起点?
wget -r --level=10 -nd http://www.mydomain.com/
More @ http://www.erichynds.com/ubuntulinux/automatically-crawl-a-website-looking-for-errors/ 更多@ http://www.erichynds.com/ubuntulinux/automatically-crawl-a-website-looking-for-errors/
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.