nodejs web scraper for password protected website

Question

I am trying to scrape a website using nodejs and it works perfectly on sites that do not require any authentication. But whenever I try to scrape a site with a form that req

Answer 1

you shouldn't use .get but .post and put the post param (username and password) in your call

request.post({
  headers: {'content-type' : 'application/x-www-form-urlencoded'},
  url:     url,
  body:    "username=myuser&password=mypw&submit=Login"
}, function(error, response, html){
    //do your parsing... 
    var $ = cheerio.load(html)
});