节点js，某些网站的请求正文为空

Question

I'm experimenting with Node.js and web scraping. In this case, I'm trying to scrape the most recent songs from a local radio station for display. With this particular website, body returns nothing. When I try using google or any other website, body has a value. Is this a feature of the website I'm trying to scrape?

Here's my code:

var request = require('request');

var url = "http://www.radiomilwaukee.org";
request(url, function(err,resp,body) {
    if (!err && resp.statusCode == 200) {
        console.log(body);
    }
    else
    {
        console.log(err);
    }

});

Answer 1

That's weird, the website you're requesting doesn't seem to return anything unless the accept-encoding header is set to gzip . With that in mind, using this gist will work: https://gist.github.com/nickfishman/5515364

I ran the code within that gist, replacing the URL with "http://www.radiomilwaukee.org" and see the content within the sample.html file once the code has completed.

If you'd rather have access to the web page's content within the code, you could do something like this:

// ...

req.on('response', function(res) {
    var body, encoding, unzipped;

    if (res.statusCode !== 200) throw new Error('Status not 200');

    encoding = res.headers['content-encoding'];
    if (encoding == 'gzip') {
        unzipped = res.pipe(zlib.createGunzip());
        unzipped.on("readable", function() {
            // collect the content in the body variable
            body += unzipped.read().toString();
        });
    }

    // ...