[英]PHP Curl following redirects
我試圖有點偷偷摸摸,作為學習過程的一部分,嘗試並提高我的頁面抓取技巧。
我遇到的一件事我還未能解決的是,某些網站會使用內部鏈接,然后重定向到外部鏈接。
我想要做的是修改一些卷曲代碼以遵循重定向,直到它們停止然后獲得最終的靜止位置URL。
有人為我推薦一些代碼嗎?
我現在有這個,但目前還沒有正確地遵循重定向。
$opts = array(CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true);
$curl = curl_init();
curl_setopt_array($curl, $opts);
$str = curl_exec($curl);
curl_close($curl);
http.//php.net/manual/en/ref.curl.php
function get_final_url( $url, $timeout = 5 )
{
$url = str_replace( "&", "&", urldecode(trim($url)) );
$cookie = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_ENCODING, "" );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
$content = curl_exec( $ch );
$response = curl_getinfo( $ch );
curl_close ( $ch );
if ($response['http_code'] == 301 || $response['http_code'] == 302)
{
ini_set("user_agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1");
$headers = get_headers($response['url']);
$location = "";
foreach( $headers as $value )
{
if ( substr( strtolower($value), 0, 9 ) == "location:" )
return get_final_url( trim( substr( $value, 9, strlen($value) ) ) );
}
}
if ( preg_match("/window\.location\.replace\('(.*)'\)/i", $content, $value) ||
preg_match("/window\.location\=\"(.*)\"/i", $content, $value)
)
{
return get_final_url ( $value[1] );
}
else
{
return $response['url'];
}
}
如果你不能使用CURLOPT_FOLLOWLOCATION
,我建議你使用像這樣的遞歸方法:
function getUrl($url, $count) {
// max number of redirects
if ($count > 5) {
return false;
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if (!$data) {
return false;
}
$dataArray = explode("\r\n\r\n", $data, 2);
if (count($dataArray) != 2) {
return false;
}
list($header, $body) = $dataArray;
if ($httpCode == 301 || $httpCode == 302) {
$matches = array();
preg_match('/Location:(.*?)\n/', $header, $matches);
if (isset($matches[1])) {
return getUrl(trim($matches[1]), $count + 1);
}
} else {
return $body;
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.