Why I get urllib2.HTTPError with urllib2 and no errors with urllib?

Question

I have the following simple code:

import urllib2
import sys
sys.path.append('../BeautifulSoup/BeautifulSoup-3.1.0.1')
from BeautifulSoup import *
page='http://en.wikipedia.org/wiki/Main_Page'
c=urllib2.urlopen(page)

This code generates the following error messages:

    c=urllib2.urlopen(page)
  File "/usr/lib64/python2.4/urllib2.py", line 130, in urlopen
    return _opener.open(url, data)
  File "/usr/lib64/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib64/python2.4/urllib2.py", line 471, in http_response
    response = self.parent.error(
  File "/usr/lib64/python2.4/urllib2.py", line 402, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 480, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

But if I replace urllib2 by urllib, I get no error messages. Can anybody explain this behavior?

Answer 1

The original urllib simply does not raise an exception on a 403 code. If you add print c.getcode() to the last line of your program, urllib will reach it and still print out 403.

Then if you do print c.read() at the end, you will see that you did indeed get an error page from Wikipedia. It's just a matter of urllib2 deciding to treat an error 403 as a runtime exception, versus urllib allowing you to still get an error 403 and then do something with the page.

Answer 2

Wikipedia seems to be filtering out urllib2's default User-Agent. Just change it.

Answer 3

Overriding urllib2.HTTPError or urllib.error.HTTPError and reading response HTML anyway this post shows some nice way to obtain detailed error message from server