Loading url with cyrillic symbols

问题

I have to load some url with cyrillic symbols. My script should work with this:

http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/

If I'll use this in browser it would replaced into normal symbols, but urllib code fails with 404 error. How to decode correctly this url?

When I'm using that url directly in code, like address = 'that address', it works perfect. But I used parsing page for getting this url. I have a list of urls which contents cyrillic. Maybe they have uncorrect encoding? Here is more code:

requestData = urllib2.Request( %SOME_ADDRESS%, None,  {"User-Agent": user_agent})
requestHandler = pageHandler.open(requestData)

pageData = requestHandler.read().decode('utf-8')
soupHandler = BeautifulSoup(pageData)

topicLinks = []
for postBlock in soupHandler.findAll('a', href=re.compile('%SOME_REGEXP%')):
    topicLinks.append(postBlock['href'])

postAddress = choice(topicLinks)

postRequestData = urllib2.Request(postAddress, None,  {"User-Agent": user_agent})
postHandler = pageHandler.open(postRequestData)
postData = postHandler.read()

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

回答1:

I have a list of urls which contents cyrillic.

OK, if it contains raw (not %-encoded) Cyrillic characters that's not like the example, and in fact it isn't a URL at all.

An address with non-ASCII characters in it is known as an IRI. IRIs shouldn't be used in an HTML link, but browsers tend to fix up these mistakes.

To convert an IRI to a URI which you can then open with urllib, you have to:

encode non-ASCII characters in the hostname part using Punycode (IDNA).
encode non-ASCII characters in rest of the IRI to UTF-8 bytes and URL-encode them (resulting in %D0%BF... like in the example URL).

an example implementation.

回答2:

You can try to use the urllib.unquote method.

>>> import urllib
>>> string = urllib.unquote("http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/")
>>> print string.decode("utf-8")
http://wincode.org/программирование/

来源：https://stackoverflow.com/questions/2835298/loading-url-with-cyrillic-symbols

标签

python

url

urllib