UnicodeError: URL contains non-ASCII characters (Python 2.7)

问题

So I've managed to make a crawler, and I'm searchng for all links and when I arrive at a product link I make some finds and I take all product information, but when it arrives to certain page it gives a unicode error :/

import urllib
import urlparse
from itertools import ifilterfalse
from urllib2 import URLError, HTTPError

from bs4 import BeautifulSoup

urls = ["http://www.kiabi.es/"]
visited = []


def get_html_text(url):
    try:
        return urllib.urlopen(current_url).read()
    except (URLError, HTTPError, urllib.ContentTooShortError):
        print "Error getting " + current_url


def find_internal_links_in_html_text(html_text, base_url):
    soup = BeautifulSoup(html_text, "html.parser")
    links = []
    for tag in soup.findAll('a', href=True):
        url = urlparse.urljoin(base_url, tag['href'])
        domain = urlparse.urlparse(base_url).hostname
        if domain in url:
            links.append(url)
    return links


def is_url_already_visited(url):
    return url in visited


while urls:
    current_url = urls.pop()
    word = '#C'
    if word in current_url:
        [do sth]
    #print "Parsing", current_url
    html_text = get_html_text(current_url)
    visited.append(current_url)
    found_urls = find_internal_links_in_html_text(html_text, current_url)
    new_urls = ifilterfalse(is_url_already_visited, found_urls)
    urls.extend(new_urls)

Error:

Traceback (most recent call last):

File "<ipython-input-1-67c2b4cf7175>", line 1, in <module>
runfile('S:/Consultas_python/Kiabi.py', wdir='S:/Consultas_python')

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile
execfile(filename, namespace)

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)

File "S:/Consultas_python/Kiabi.py", line 91, in <module>
html_text = get_html_text(current_url)

File "S:/Consultas_python/Kiabi.py", line 30, in get_html_text
return urllib.urlopen(current_url).read()

File "C:\Anaconda2\lib\urllib.py", line 87, in urlopen
return opener.open(url)

File "C:\Anaconda2\lib\urllib.py", line 185, in open
fullurl = unwrap(toBytes(fullurl))

File "C:\Anaconda2\lib\urllib.py", line 1070, in toBytes
" contains non-ASCII characters")

UnicodeError: URL u'http://www.kiabi.es/Barbapap\xe1_s1' contains non-ASCII characters

UnicodeError: URL u'http://www.kiabi.es/Petit-B\xe9guin_s2' contains non-ASCII characters

How can I fix it?

回答1:

You need to percent encode the utf8 representation of your unicode string.

As explained here:

All non-ASCII code points in the IRI should next be encoded as UTF-8, and the resulting bytes percent-encoded, to produce a valid URI.

In python code, that means:

import urllib
url = urllib.quote(url.encode('utf8'), ':/')

The second argument to quote, ':/', is to prevent the colon in the protocol part http:, or path separator / from being encoded.

(In Python 3, the quote function has been moved to the urllib.parse module).

回答2:

You can try to encode the urls. Your code may look like:

def get_html_text(url):
    try:
        return urllib.urlopen(current_url.encode('ascii','ignore')).read()
    except (URLError, HTTPError, urllib.ContentTooShortError):
        print "Error getting " + current_url

来源：https://stackoverflow.com/questions/33708059/unicodeerror-url-contains-non-ascii-characters-python-2-7

标签

python

unicode

beautifulsoup

web-crawler

non-ascii-characters