Python script receiving a UnicodeEncodeError: 'ascii' codec can't encode character

Question

I have a simple Python script that pulls posts from reddit and posts them on Twitter. Unfortunately, tonight it began having issues that I\'m assuming are because of someone\'s

Answer 1

The problem likely arises from mixing bytestrings and unicode strings on concatenation. As an alternative to prefixing all string literals with u, maybe

from __future__ import unicode_literals

fixes things for you. See here for a deeper explanation and to decide whether it's an option for you or not.

Answer 2

You are trying to print a unicode string to your terminal (or possibly a file by IO redirection), but the encoding used by your terminal (or file system) is ASCII. Because of this Python attempts to convert it from the unicode representation to ASCII, but fails because codepoint u'\u201c' (“) can not be represented in ASCII. Effectively your code is doing this:

>>> print u'\u201c'.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 0: ordinal not in range(128)

You could try converting to UTF-8:

print (post + " " + post_dict[post] + " #python").encode('utf8')

or convert to ASCII like this:

print (post + " " + post_dict[post] + " #python").encode('ascii', 'replace')

which will replace invalid ASCII characters with ?.

Another way, which is useful if you are printing for debugging purposes, is to print the repr of the string:

print repr(post + " " + post_dict[post] + " #python")

which would output something like this:

>>> s = 'string with \u201cLEFT DOUBLE QUOTATION MARK\u201c'
>>> print repr(s)
u'string with \u201cLEFT DOUBLE QUOTATION MARK\u201c'

Answer 3

Consider this simple program:

print(u'\u201c' + "python")

If you try printing to a terminal (with an appropriate character encoding), you get

“python

However, if you try redirecting output to a file, you get a UnicodeEncodeError.

script.py > /tmp/out
Traceback (most recent call last):
  File "/home/unutbu/pybin/script.py", line 4, in <module>
    print(u'\u201c' + "python")
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 0: ordinal not in range(128)

When you print to a terminal, Python uses the terminal's character encoding to encode unicode. (Terminals can only print bytes, so unicode must be encoded in order to be printed.)

When you redirect output to a file, Python can not determine the character encoding since files have no declared encoding. So by default Python2 implicitly encodes all unicode using the ascii encoding before writing to the file. Since u'\u201c' can not be ascii encoded, a UnicodeEncodeError. (Only the first 127 unicode code points can be encoded with ascii).

This issue is explained in detail in the Why Print Fails wiki.

---

To fix the problem, first, avoid adding unicode and byte strings. This causes implicit conversion using the ascii codec in Python2, and an exception in Python3. To future-proof your code, it is better to be explicit. For example, encode post explicitly before formatting and printing the bytes:

post = post.encode('utf-8')
print('{} {} #python'.format(post, post_dict[post]))