request() function returns undefined values

Question

so I'm currently making a Google Chrome Extension that will notify me whenever a new grade gets posted to my University's gradebook for all of my classes, so currently I'm trying to iteratively crawl and scrape the URL's and compare it to the last iteration (...? advice on this would be appreciated!), and currently when I use the request() function, even with asynchronization, the function currently returns undefined for the response and body, and gives me this other weird thing for the error if I try to console.log all of those.

Here's the error I'm getting after that:

bundle.js:24 Uncaught TypeError: Cannot read property 'headers' of undefined
    at Request._callback (bundle.js:24)
    at self.callback (bundle.js:54273)
    at Request.EventEmitter.emit (bundle.js:95413)
    at Request.start (bundle.js:54842)
    at Request.end (bundle.js:55610)
    at end (bundle.js:54652)
    at bundle.js:54666
    at Item.run (bundle.js:103974)
    at drainQueue (bundle.js:103944)

And here's my code (changed the URL so you don't see my school's login URL):

var Crawler = require("simplecrawler"),
    url = require("url"),
    cheerio = require("cheerio"),
    request = require("request");

var initialURL = "https://www.fakeURL.com/";


var crawler = new Crawler(initialURL);

request("https://www.fakeURL.com/", {
    // The jar option isn't necessary for simplecrawler integration, but it's
    // the easiest way to have request remember the session cookie between this
    // request and the next
    jar: true,
    mode: 'no-cors'
}, function(error, response, body) {
    // Start by saving the cookies. We'll likely be assigned a session cookie
    // straight off the bat, and then the server will remember the fact that
    // this session is logged in as user "iamauser" after we've successfully
    // logged in

    crawler.cookies.addFromHeaders(response.headers["set-cookie"]);

    // We want to get the names and values of all relevant inputs on the page,
    // so that any CSRF tokens or similar things are included in the POST
    // request
    var $ = cheerio.load(body),
        formDefaults = {},
        // You should adapt these selectors so that they target the
        // appropriate form and inputs
        formAction = $("#login").attr("action"),
        loginInputs = $("input");

    // We loop over the input elements and extract their names and values so
    // that we can include them in the login POST request
    loginInputs.each(function(i, input) {
        var inputName = $(input).attr("name"),
            inputValue = $(input).val();

        formDefaults[inputName] = inputValue;
    });

    // Time for the login request!
    request.post(url.resolve(initialURL, formAction), {
        // We can't be sure that all of the input fields have a correct default
        // value. Maybe the user has to tick a checkbox or something similar in
        // order to log in. This is something you have to find this out manually
        // by logging in to the site in your browser and inspecting in the
        // network panel of your favorite dev tools what parameters are included
        // in the request.
        form: Object.assign(formDefaults, {
            username: "secretusername",
            password: "secretpassword"
        }),
        // We want to include the saved cookies from the last request in this
        // one as well
        jar: true
    }, function(error, response, body) {
        // That should do it! We're now ready to start the crawler
        crawler.interval = 10000 //600000 // 10 minutes
        crawler.maxConcurrency = 1; // 1 active check at a time
        crawler.maxDepth = 5;
        crawler.start();
    });
});

crawler.on("fetchcomplete", function(queueItem, responseBuffer, response) {
    console.log("Fetched", queueItem.url, responseBuffer.toString());
});

// crawler.interval = 600000 // 10 minutes
// crawler.maxConcurrency = 1; // 1 active check at a time
// crawler.maxDepth = 5;
//
// crawler.start();

One thing to note is I added the 'no-cors' mode to my request so I can stop having problems with CORS whenever I tested this, but is that possibly what's causing this issue?

Thanks!

EDIT: I'm using Browserify to use the require() stuff in the browser. I can't post the actual code from bundle.js because it is extremely long and won't fit in here. Just wanted to clarify that. Thanks!

EDIT2: Here is what I'm given when I try to do console.log(error):

Error: Invalid value for opts.mode
    at new module.exports (bundle.js:108605)
    at Object.http.request (bundle.js:108428)
    at Object.https.request (bundle.js:97056)
    at Request.start (bundle.js:54843)
    at Request.end (bundle.js:55613)
    at end (bundle.js:54655)
    at bundle.js:54669
    at Item.run (bundle.js:103977)
    at drainQueue (bundle.js:103947)

Answer 1

As James said, if you get an error, check the error by logging it to console or whatever method you like most to show debug info.

If you are getting Cannot read property 'headers' of undefined is, as you said, that response is undefined, so your first callback line will fail, because it tries to access response.headers .

The way to simple-debug here is to console.log() the error, before reaching the line with the problem (as it halts there), so you have to simply add console.log(error); as the first line of your callback.

The way to go:

Although you may fix the problems you see in your console.log(error) , this code is doomed because you are not checking if you received an error and assuming that the request filled successfully. A net connection is chaotic, and a request can fail for a lot of reasons, so before accessing request.headers you have to check if any error happened and log it (or show it to you client, retry the request after X seconds, whatever you like most).

Tip : If you have a callback with an error parameter, check it. Is there as the first parameter for a reason.

The code will look like this:

request("https://www.fakeURL.com/", {
    jar: true,
    mode: 'no-cors'
}, function(error, response, body) {
    if (error) {
        console.log(error);
        makeTheRequestAgainIn(5000); // Milliseconds
    } else {
        doWhateverWith(response, body);
    }
});

The error:

Simply you can't disable CORS in the browser. You can disable it in node.js, as is not a browser, that's why there's a option for that in the request module, but browsers have security measures for a reason. If they can be avoided then they make no sense.

In short: Yes, you are having problems with CORS if you don't have enabled in the server.

Protip : When dealing with JavaScript in the browser, is a good practice to have the developer tools open (F12), as you have, this way you had seen the CORS error automatically logged in the console (or whatever network error happens). Also, switching to the Network tab and checking the request headers, response and such is a good practice too.

EDIT: Just noticed the Chrome Extension thing (dang). Extensions are less limited, so is fine to do those calls, as you can read here: https://developer.chrome.com/extensions/xhr

Also, checked the request npm module source code and do not have a no-cors value. I think that you mixed the Request API with the request module.