unicode().decode('utf-8', 'ignore') raising UnicodeEncodeError

Question

Here is the code:

>>> z = u\'\\u2022\'.decode(\'utf-8\', \'ignore\')
Traceback (most recent call last):
  File \"\", line 1, in 

        

Answer 1

When I first started messing around with python strings and unicode, It took me awhile to understand the jargon of decode and encode too, so here's my post from here that may help:

---

Think of decoding as what you do to go from a regular bytestring to unicode and encoding as what you do to get back from unicode. In other words:

You de - code a str to produce a unicode string

and en - code a unicode string to produce an str.

So:

unicode_char = u'\xb0'

encodedchar = unicode_char.encode('utf-8')

encodedchar will contain your unicode character, displayed in the selected encoding (in this case, utf-8).

Answer 2

You're trying to decode a unicode. The implicit encoding to make the decode work is what's failing.

Answer 3

From http://wiki.python.org/moin/UnicodeEncodeError

Paradoxically, a UnicodeEncodeError may happen when decoding. The cause of it seems to be the coding-specific decode() functions that normally expect a parameter of type str. It appears that on seeing a unicode parameter, the decode() functions "down-convert" it into str, then decode the result assuming it to be of their own coding. It also appears that the "down-conversion" is performed using the ASCII encoder. Hence an encoding failure inside a decoder.