由于SSL错误，无法抓取HTML网站

Question

I am working on a scraping script. It works on most websites but I cannot access a specific SSL site.

Here is my code:

if (!extension_loaded('openssl')){
    // not occurring
}

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.chase.com/');
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);

$result = curl_exec($ch);

if($result === false)
{
    $err = curl_error($ch);
    //$err = SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054
}

$result is always FALSE , and it shows this error message:

SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054

But it works on other websites that have SSL. I also checked phpinfo() , cURL and OpenSSL are active. I am using WAMP, any ideas?

Answer 1

You need to set a Useragent. I tested with and without one and it fixes the issue. It appears Chase is wanting a UA to be provided in the request.

So add this:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US)'); 

Answer 2

I solved issue by just using following PHP librery.

https://github.com/rmccue/Requests

[use this library code on your Linux based server, may be it will not work on xampp or wamp ]