Changing user agent on urllib2.urlopen

Question

How can I download a webpage with a user agent other than the default one on urllib2.urlopen?

Answer 1

headers = { 'User-Agent' : 'Mozilla/5.0' }
req = urllib2.Request('www.example.com', None, headers)
html = urllib2.urlopen(req).read()

Or, a bit shorter:

req = urllib2.Request('www.example.com', headers={ 'User-Agent': 'Mozilla/5.0' })
html = urllib2.urlopen(req).read()

Answer 2

For python 3, urllib is split into 3 modules...

import urllib.request
req = urllib.request.Request(url="http://localhost/", headers={'User-Agent':' Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0'})
handler = urllib.request.urlopen(req)

Answer 3

For urllib you can use:

from urllib import FancyURLopener

class MyOpener(FancyURLopener, object):
    version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'

myopener = MyOpener()
myopener.retrieve('https://www.google.com/search?q=test', 'useragent.html')