[英]Crawling Google Search with PHP
I trying to get my head around how to fetch Google search results with PHP or JavaScript. 我试图了解如何使用PHP或JavaScript获取Google搜索结果。 I know it has been possible before but now I can't find a way.
我知道以前有可能,但现在我找不到方法。
I am trying to duplicate (somewhat) the functionality of 我试图复制(有点)的功能
http://www.getupdated.se/sokmotoroptimering/seo-verktyg/kolla-ranking/ http://www.getupdated.se/sokmotoroptimering/seo-verktyg/kolla-ranking/
But really the core issue I want to solve is just to get the search result via PHP or JavaScript,the rest i can figure out. 但实际上我想要解决的核心问题只是通过PHP或JavaScript获取搜索结果,其余的我可以搞清楚。
Fetching the results using file_get_contents() or cURL doesn't seem to work. 使用file_get_contents()或cURL获取结果似乎不起作用。
Example: 例:
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/#hl=sv&q=dogs');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$result = curl_exec($ch);
curl_close($ch);
echo '<pre>';
var_dump($result);
echo '</pre>';
Results: 结果:
string(219) "302 Moved The document has moved here."
string(219)“302已移动文档已移至此处。”
So, with some Googling i found http://code.google.com/apis/customsearch/v1/overview.html but that seems to only work for generating a custom search for one or more websites. 因此,通过一些谷歌搜索,我找到了http://code.google.com/apis/customsearch/v1/overview.html,但这似乎只适用于为一个或多个网站生成自定义搜索。 It seem to require a "Custom Search Engine" cx-parameter passed.
它似乎需要传递“自定义搜索引擎”cx参数。
So anyway, any idea? 无论如何,任何想法?
I did it earlier. 我之前做过。 Generate the html contents by making
https://www.google.co.in/search?hl=en&output=search&q=india
http request, now parse specific tags using the htmldom php library. 通过
https://www.google.co.in/search?hl=en&output=search&q=india
http请求生成html内容,现在使用htmldom php库解析特定标记。 You can parse the content of result page using PHP SIMPLE HTML DOM 您可以使用PHP SIMPLE HTML DOM解析结果页面的内容
DEMO : Below code will give you title of all the result :
演示:下面的代码将为您提供所有结果的标题:
<?php
include("simple_html_dom.php");
$html = file_get_html('http://www.google.co.in/search?hl=en&output=search&q=india');
$i = 0;
foreach($html->find('li[class=g]') as $element) {
foreach($element->find('h3[class=r]') as $h3)
{
$title[$i] = '<h1>'.$h3->plaintext.'</h1>' ;
}
$i++;
}
print_r($title);
?>
There is php a github package named google-url that does the job. 有一个名为google-url的github包可以完成这项工作。
The api is very comfortable to use. api使用起来非常舒服。 See the example :
看例子:
// this line creates a new crawler
$googleUrl=new \GoogleURL\GoogleUrl();
$googleUrl->setLang('en'); // say for which lang you want to search (it could have been "fr" instead)
$googleUrl->setNumberResults(10); // how many results you want to check
// launch the search for a specific keyword
$results = $googleUrl->search("google crawler");
// finaly you can loop on the results (an example is also available on the github page)
However you will have to think to use a delay between each query, or else google will consider you as a bot and ask you for a captcha that will lock the script. 但是,您必须考虑在每个查询之间使用延迟,否则Google会将您视为机器人并要求您提供将锁定脚本的验证码。
Odd. 奇。 Because if I do a
curl
from the command like I get a 200 OK
: 因为如果我从命令做一个
curl
,就像我得到200 OK
:
curl -I 'http://www.google.se/#hl=sv&q=dogs'
HTTP/1.1 200 OK
Date: Sun, 27 Jan 2013 20:45:02 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=b82cb66e9d996c48:FF=0:TM=1359319502:LM=1359319502:S=D-LW-_w8GlMfw-lX; expires=Tue, 27-Jan-2015 20:45:02 GMT; path=/; domain=.google.se
Set-Cookie: NID=67=XtW2l43TDBuOaOnhWkQ-AeRbpZOiA-UYEcs7BIgfGs41FkHlEegssgllBRmfhgQDwubG3JB0s5691OLHpNmLSNmJrKHKGZuwxCJYv1qnaBPtzitRECdLAIL0oQ0DSkrx; expires=Mon, 29-Jul-2013 20:45:02 GMT; path=/; domain=.google.se; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
Also, maybe consider setting a urlencode
for the passed URL so this line: 另外,也可以考虑为传递的URL设置
urlencode
,以便这一行:
curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/#hl=sv&q=dogs');
Changes to this: 对此的更改:
curl_setopt($ch, CURLOPT_URL, 'http://www.google.se/' . urlencode('#hl=sv&q=dogs'));
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.