Again urllib.error.HTTPError: HTTP Error 400: Bad Request

Question

Hy! I tried to open web-page, that is normally opening in browser, but python just swears and does not want to work.

import urllib.request, urllib.error
f =          


        

Answer 1

This URL seems to be doing user agent string checking. If I adjust my user agent string in Firefox to Python-urllib/2.7, it fails with the Bad Request you are seeing.

As you are using urllib, you can adjust the User Agent following this tutorial

from urllib.request import FancyURLopener

class MyOpener(FancyURLopener):
    version = 'My new User-Agent'   # Set this to a string you want for your user agent

myopener = MyOpener()
page = myopener.open('http://www.booking.com/reviewlist.html?cc1=tr;pagename=sapphire')

Answer 2

They are probably blocking the fact that it isn't coming from a browser. You probably need a valid User-Agent header or something.

Using requests, this works:

import requests
headers = 
{
 'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)     Chrome/37.0.2049.0 Safari/537.36'
}

r = requests.get('http://www.booking.com/reviewlist.html?cc1=tr;pagename=sapphire', headers=headers)
print r
print r.headers