How do you handle cookies with request-promise?

Question

I'm having trouble scraping a website that needs authentication, and is using session cookies. The session requires a request with POST, and the authentication then approves. But when I want to GET the webpage that need authentication, it returns "Unauthorized". I guess I need a way to bring the session cookie with the GET-request, but I don't know how! My dependencies is request-promise( https://www.npmjs.com/package/request-promise ).

The code looks like this:

var rp = require("request-promise");

    var options = {
    method: "POST",
    uri: "http://website.com/login",
    form: {
        username: "user",
        password: "pass",
    },
    headers: {},
    simple: false
};

rp(options).then(function(response) {
    console.log(response); // --> "Redirecting to login/AuthPage"
    request("http://website.com/login/AuthPage", function(err, res, body) {
        console.log(body); // --> "Unauthorized"
    })
}).catch(function(e) {
    console.log(e)
})

I'm guessing you have to put the request in a "Jar" ( https://github.com/request/request#requestjar ), to be able to reach the next request-URL, but how can I set the request-promise to create a cookie-jar?

Answer 1

Your problem is how to keep the session after authentication. That means, after logging in by using username and password, the server will return a cookie with an identifier. Then you need to attach that cookie to all your feature requests.
It's simple with request-promise . Just keep tracking session by enabling jar option then use the same request object for all requests. Let take a look

var request = require("request-promise").defaults({ jar: true });
var options = {
    method: "POST",
    uri: "http://website.com/login",
    form: {
        username: "user",
        password: "pass",
    },
    headers: {},
    simple: false
};

request(options).then(function(response) {
    request("http://website.com/login/AuthPage", function(err, res, body) {
        console.log(body);
    })
}).catch(function(e) {
    console.log(e)
})

Answer 2

Use the following object while making rest calls.

var request = require("request-promise").defaults({jar: true});

To add your own cookies

var tough = require('tough-cookie');

// Easy creation of the cookie - see tough-cookie docs for details
let cookie = new tough.Cookie({
    key: "some_key",
    value: "some_value",
    domain: 'api.mydomain.com',
    httpOnly: true,
    maxAge: 31536000
});

// Put cookie in an jar which can be used across multiple requests
var cookiejar = rp.jar();
cookiejar.setCookie(cookie, 'https://api.mydomain.com');
// ...all requests to https://api.mydomain.com will include the cookie

var options = {
    uri: 'https://api.mydomain.com/...',
    jar: cookiejar // Tells rp to include cookies in jar that match uri
};

and then make the call. More details about request-promise : https://www.npmjs.com/package/request-promise