UnicodeEncodeError: 'ascii' codec can't encode character

Question

When uploading files with non-ASCII characters I get UnicodeEncodeError:

Exception Type: UnicodeEncodeError at /admin/studio/newsitem/add/
Exception Value: \         


        

Answer 1

akaihola's answer was helpful. For those who run django app with uWSGI managed via upstart script, just add these lines to your /etc/init/yourapp.conf

env LANG="en_US.utf8"
env LC_ALL="en_US.UTF-8"
env LC_LANG="en_US.UTF-8"

It solved the problem for me.

Answer 2

As said before, it is related to locale. For exemple, if you use gunicorn to serve your django application, you may have an init.d script (or, as me, a runit script), where you can set the locale.

To solve UnicodeEncodeError with file upload, put something like export LC_ALL=en_US.UTF8 in your script that run your app.

For example, this is mine (using gunicorn and runit):

#!/bin/bash
export LC_ALL=en_US.UTF8
cd /path/to/app/projectname
exec gunicorn_django -b localhost:8000 --workers=2

Also, you can check your locale in your template, using this in your view:

import locale
data_to_tpl = {'loc': locale.getlocale(), 'lod_def': locale.getdefaultlocale()}

And just disply {{loc}} - {{loc_def}} in your template.

You will have more information about your locale settings! That was very usefull for me.

Answer 3

Another useful option that avoids rewriting code is to change the default encoding for python.

If you're using virtualenv you can change (or create if doesn't exist) env/lib/python2.7/sitecustomize.py and add:

import sys
sys.setdefaultencoding('utf-8')

or, if you are in a production system, you can do the same to /usr/lib/python2.7/sitecustomize.py

Answer 4

In situations where you must display a unicode string in a place that only accepts ascii (like the console or as a path) you must tell Python that you want it to replace the non ascii characters best effort.

>> problem_str = u'This is not all ascii\xf8 man'
>> safe_str = problem_str.encode('ascii', 'ignore')
>> safe_str
'This is not all ascii man'

Encoding issues are prevented in the admin by the cautious handing of Django templating, but if you have ever added custom columns and forgotten to convert the values to ascii, or you override the str method for a model and forget to do this, you will get the same error, preventing template rendering.

If this string were saved into your (hopefully utf8) database there would be no problem, it looks like you are trying to upload a file that uses the title of an entity that has a non ascii character.

Answer 5

Using python 2.7.8 and Django 1.7, I solved my problem by importing:

from __future__ import unicode_literals

and using force_text():

from django.utils.encoding import force_text

Answer 6

It's hard to say without seeing a little more code but it looks to be related to this question: UnicodeDecodeError on attempt to save file through django default filebased backend.

Looking through the Django ticket mentioned it would seem you should follow something similar to the deployment docs on "If you get a UnicodeEncodeError":
https://docs.djangoproject.com/en/1.4/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror

(I know this is for Apache/mod_python but my guess would be it's the same root issue of file system encoding not being UTF-8 and there is a similar fix when using nginx)

EDIT: From what I can tell this nginx module would be the equivalent fix: http://wiki.nginx.org/NginxHttpCharsetModule