使用NPM搜寻器搜寻Node.js错误

Question

When using the npm crawler to crawl links, I get the error:

C:\Users\ryani\Desktop\JavaScript\crawler\crawler.js:15
                                $('a').each(function(index, value) {
                                      ^

TypeError: Cannot read property 'each' of undefined

I've tried setting timeouts and various debugging techniques... I'm not sure why it is getting undefined; when I put the code in a tag on an html page, it works fine.

crawler.js:

var Crawler = require("crawler");

var c = new Crawler({
    "maxConnections":10,

    "callback":function(error, res, $) {

        if (error) {
            console.log("error");
            console.log(error);
        } else {
            $('a').each(function(index, value) {
                console.log($(this).attr('href'));
                //c.queue(href)
            });
        }
    }
});

c.queue('http://www.google.com');

Answer 1

The problem is that you have not initialized Cheerio (var $ = res.$;).

Try this one it fetches all the likes from the provided URL...

var Crawler = require("crawler");

var c = new Crawler({
    maxConnections: 10,
    // This will be called for each crawled page 
    callback: function(error, res, done) {
        if (error) {
            console.log(error);
        } else {
            var $ = res.$;
            var links = [];

            $('a').each(function(i, elem) {
                links[i] = $(this).attr('href');
            });
            // $ is Cheerio by default 
            //a lean implementation of core jQuery designed specifically for the server 
            console.log(links);
        }
        done();
    }
});

c.queue('http://www.google.com');

Answer 2

I have never used the Crawler node module before but looking at their usage example found here .

The 3 parameters for the callback function are;

1. error - The potential error that might returned by the crawler API
2. res - The response object and $ is 1 of its property
3. done - another callback function which the code will call when its done processing.

By writing your code as callback :function(error, res, $) { the $ variable is in the 3rd position of parameter so essentially it is been used to represent as the done function. Now you're saying done . $ which is incorrect because the function object does not have that property and hence the error.

That is, your code should look something like:

    res.$('a').each(function(i, elem) {
        links[i] = $(this).attr('href');
    });

Also, you will need to call the done parameter otherwise the process will just stuck there. eg $() . However I recommend you follow their code example as $ is not a good variable name. Recommend renaming your 3rd variable $ as done instead.