使用Curl抓取网站返回空白结果

Question

What i'm trying to do is do a search on Amazon using a random keyword, then i'll just scrape maybe the first 10 results, the issue when i print the html results i get nothing, it's just blank, my code looks ok to me and i have used CURL in the past and never come accross this, my code:

<?php

include_once("classes/simple_html_dom.php");

function get_random_keyword() {
    $f_contents = file("keywords.txt"); 
    return $f_contents[rand(0, count($f_contents) - 1)];    
}

function getHtml($page) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $page);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5');
    $html = curl_exec($ch);
    print "html -> " . $html;
    curl_close($ch);    
    return $html;
}


$html = getHtml("https://www.amazon.co.uk/s?k=" . get_random_keyword());

?>

Ideally i would have preferred to use the API, but from what i understand you need 3 sales first before you are granted access, can anyone see any issues? i'm not sure what else to check, any help is appreciated.

Answer 1

Amazon is returning the response encoded in gzip. You need to decode it:

$html = getHtml("https://www.amazon.co.uk/s?k=" . get_random_keyword());
echo gzdecode($html);