How to maintain a request session in NodeJS

Question

I\'m trying to use NodeJS to scrape a website that requires a login by POST. Then once I\'m logged in I can access a separate webpage by GET<

Answer 1

The request.jar(); didn't work for me. So I am using the headers response to make another request like this:

request.post({
    url: 'https://exampleurl.com/login',
    form: {"login":"xxxx", "password":"xxxx"}
}, function(error, response, body){

    request.get({
        url:"https://exampleurl.com/logged",
        header: response.headers
    },function(error, response, body){
        // The full html of the authenticated page
        console.log(body);
    });
});

Actualy this way is working fine. =D

Answer 2

You need to make a cookie jar and use the same jar for all related requests.

 var cookieJar = request.jar();
 request.post({url : requesturl, jar: cookieJar, form: lform}, ...

That should in theory allow you to scrape pages with GET as a logged-in user, but only once you get the actual login code working. Based on your description of the response to your login POST, that may not be actually working correctly yet, so the cookie jar won't help until you fix the problems in your login code first.

Answer 3

Request manages cookies between requests if you enable it:

Cookies are disabled by default (else, they would be used in subsequent requests). To enable cookies, set jar to true (either in defaults or options).

const request = request.defaults({jar: true})
request('http://www.google.com', function () {
  request('http://images.google.com')
});