UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

Question

I\'m having problems dealing with unicode characters from text fetched from different web pages (on different sites). I am using BeautifulSoup.

The problem is that

Answer 1

In shell:

1. Find supported UTF-8 locale by the following command:

   locale -a | grep "UTF-8"
   

2. Export it, before running the script, e.g.:

   export LC_ALL=$(locale -a | grep UTF-8)
   

   or manually like:

   export LC_ALL=C.UTF-8
   

3. Test it by printing special character, e.g. ™:

   python -c 'print(u"\u2122");'
   

Above tested in Ubuntu.