[英]php curl working on one localhost but not on other
我從我的客戶那里得到了一個使用 php curl 抓取網站的命令。 我完成了這項工作,腳本在我的本地主機上運行良好。 但是當我把它交給我的客戶端腳本時,他的本地主機上沒有工作。
<?php
ini_set('display_errors', 'On');
error_reporting(E_ALL);
print "Cascading https://www.autotrader.ca/cars/on/toronto/?rcp=15&rcs=0&prx=100&prv=Ontario&loc=toronto%2C%20on&hprc=True&wcp=True&sts=New-Used&inMarket=basicSearch&mdl=Accent&make=Hyundai&scrladid=11543266:<p>";
$array = [];
$array[] = "/a/hyundai/accent/oshawa/ontario/19_11543266_/?showcpo=ShowCpo&ncse=no&orup=1_15_340&sprx=100";
$array[] = "/a/hyundai/accent/cambridge/ontario/5_48590586_20200220145456261/?showcpo=ShowCpo&ncse=no&orup=2_15_340&sprx=100";
$array[] = "/a/hyundai/accent/mississauga/ontario/19_11536424_/?showcpo=ShowCpo&ncse=no&orup=3_15_340&sprx=100";
foreach ($array as $key=>$value)
{
$scrape = "https://www.autotrader.ca".$array[$key];
print "Scraping $scrape<p>";
echo "<br>";
$user_agent = 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Mobile Safari/537.36';
$headers = [
'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding: gzip, deflate, br',
'accept-language: en-US,en;q=0.9',
'cache-control: max-age=0',
'sec-fetch-dest: document',
'sec-fetch-mode: navigate',
'sec-fetch-site: none',
'sec-fetch-user: ?1',
'upgrade-insecure-requests: 1',
'user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Mobile Safari/537.36',
];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $scrape);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_TIMEOUT, 100);
curl_setopt($ch, CURLOPT_ENCODING, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_VERBOSE, true);
// curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Length: 0'));
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__) . '/cookie.txt');
$contents = curl_exec($ch);
if ($contents === FALSE){
echo "Error : ".curl_error($ch);
echo "<br>";
print "contents returned for $key = FALSE<br>";
}
curl_close($ch);
// echo $contents;
$start_pos = strpos($contents, "<title>", 0);
$end_pos = strpos($contents, "</title>", 0);
$title = substr($contents, $start_pos+7, $end_pos-$start_pos);
print "Listing $key: $title<p>";
echo "<br>";
echo "<br>";
}
他還告訴他,在沒有使用 curl 而是使用任何其他方法之前,他正在抓取網站,他認為他們已將他的請求限制在他們的服務器上,但請注意,他仍然可以在瀏覽器中訪問該網站。 如果他在 curl 中用 google url 替換 url,我檢查了他是否能夠得到正確的響應。
這里最可能的問題是您的客戶端安裝的 PHP 沒有安裝或啟用 php-curl 擴展。 這取決於您的操作系統以及 PHP 的安裝方式而有所不同,但這里有一些常見情況:
對於Ubuntu或其他基於 Debian 的 Linux分布:
apt-get install php7.4-curl
systemctl restart apache2
將“7.4”替換為您當前在第一個命令中使用的 PHP 版本
對於 Windows 上的 WAMP : 如何在 Wamp 服務器中啟用 curl
For XAMPP on Windows : How to enable cURL in PHP / XAMPP
在代理后面運行它,工作正常。 簡化並糾正了一些小錯誤。
試試這個,不要忘記評論/編輯 CURLOPT_PROXY 行。
<?php
ini_set('display_errors', 'On');
error_reporting(E_ALL);
$array = [
"/a/hyundai/accent/oshawa/ontario/19_11543266_/?showcpo=ShowCpo&ncse=no&orup=1_15_340&sprx=100",
"/a/hyundai/accent/cambridge/ontario/5_48590586_20200220145456261/?showcpo=ShowCpo&ncse=no&orup=2_15_340&sprx=100",
"/a/hyundai/accent/mississauga/ontario/19_11536424_/?showcpo=ShowCpo&ncse=no&orup=3_15_340&sprx=100"
];
foreach ($array as $key => $value) {
$scrape = "https://www.autotrader.ca" . $value;
echo "Scraping " . $scrape . "<br>\n";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $scrape);
curl_setopt($ch, CURLOPT_PROXY, "http://<proxy_url>:80"); // Comment if not behind a proxy
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__) . '/cookie.txt');
$contents = curl_exec($ch);
if (curl_error($ch)) {
echo "Error : " . curl_error($ch) . "<br>\n";
break;
}
curl_close($ch);
$title = explode("<title>", $contents);
$title = explode("</title>", $title[1]);
$title = $title[0];
echo "Listing " . $key . ": " . $title . "<br>\n";
echo "<br>\n";
echo "<br>\n";
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.