Submitting to a web form using python

Question

I have seen questions like this asked many many times but none are helpful

Im trying to submit data to a form on the web ive tried requests, and urllib and none have

Answer 1

import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
      'location' : 'Northampton',
      'language' : 'Python' }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req) 
the_page = response.read()

This makes a POST request with the data specified in the values. we need urllib to encode the url and then urllib2 to send a request.

Answer 2

If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs):

r = requests.get('http://stackoverflow.com', params=data)

This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn't what you are looking for.

You really want to POST to a form. Try this:

r = requests.post('https://stackoverflow.com/search', data=data)

This is essentially the same as GET-ting https://stackoverflow.com/questions/tagged/python , but I think you'll get the idea from this.

Answer 3

Mechanize library from python is also great allowing you to even submit forms. You can use the following code to create a browser object and create requests.

import mechanize,re
br = mechanize.Browser()
br.set_handle_robots(False)   # ignore robots
br.set_handle_refresh(False)  # can sometimes hang without this
br.addheaders = [('User-agent', 'Firefox')]             
br.open( "http://google.com" )
br.select_form( 'f' )
br.form[ 'q' ] = 'foo'
br.submit()
resp = None

for link in br.links():
    siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )

    if siteMatch:
        resp = br.follow_link( link )
        break

content = resp.get_data()
print content