Programmatically logging into a site

Question

It might sound silly; but can we programmatically login into a site such as Linkedin by passing our user credentials (userid and password)? I am not talking about using OAuth or

Answer 1

Yes, you could use a HttpWebRequest in order to send HTTP requests to a given URL. So the idea would be to POST your credentials to the given url which is supposed to authenticate the user and capture any cookies that could be used on subsequent requests to authenticated parts of the site.

Of course that's pretty raw work, if the remote site provides an API you might also use this API as it will simplify things.

Answer 2

You can use tools like selenium to automate and generate code for programmatically login. Also a search in google sure reveals more tools aimed to do "mashups" of web sites, like BeautifulSoup recommended in first answer.

Regards

Answer 3

You can login to many sites this way, using scripting. I generally prefer Python with BeautifulSoup, but there are many other possibilities for languages and libraries.

The general idea is to locate the code for the form that has the username and password fields, fill them in, then post the form.

This works for me on Linux, with my LinkedIn username and password in my .netrc in the standard format:

#!/usr/bin/python
import sys, os, urllib, urllib2, cookielib, urlparse, netrc
from BeautifulSoup import BeautifulSoup
URL = 'https://www.linkedin.com/uas/login'
COOKIES = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
HANDLER = getattr(urllib2, urlparse.urlsplit(URL).scheme.upper() + 'Handler')
DEBUGLEVEL = 0  # set to 1 to see unencrypted data
OPENER = urllib2.build_opener(COOKIES, HANDLER(debuglevel = DEBUGLEVEL))
def fetch(url, data = None):
 connection = OPENER.open(url, data)
 page = connection.read()
 connection.close()
 return page
def login():
 soup = BeautifulSoup(fetch(URL))
 form = soup.find('form')
 fields = form.findAll('input')
 auth_info = netrc.netrc().authenticators(urlparse.urlsplit(URL).netloc)
 formdata = dict([[field['name'], field['value']] for field in fields])
 formdata['session_key'], ignored, formdata['session_password'] = auth_info
 assert form['method'] == 'POST'
 posturl = urlparse.urljoin(URL, form['action'])
 print fetch(posturl, urllib.urlencode(formdata))
if __name__ == '__main__':
 login()