UnicodeEncodeError: 'ascii' codec can't encode character

Question

When uploading files with non-ASCII characters I get UnicodeEncodeError:

Exception Type: UnicodeEncodeError at /admin/studio/newsitem/add/
Exception Value: 'ascii' codec can't encode character u'\xf8' in position 78: ordinal not in range(128)

See full stack trace.

I run Django 1.2 with MySQL and nginx and FastCGI.

This is a problem that is fixed according to the Django Trac database, but I still have the problem. Any suggestions on how to fix are welcome.

EDIT: This is my image field:

image = models.ImageField(_('image'), upload_to='uploads/images', max_length=100)

Answer 1

For anyone encountering this problem when running Django with Supervisor, the solution is to add e.g. the following to the supervisord section of Supervisor's configuration:

environment=LANG="en_US.utf8", LC_ALL="en_US.UTF-8", LC_LANG="en_US.UTF-8"

This solved the problem for me in Supervisor 3.0a8 running on Debian Squeeze.

Answer 2

In situations where you must display a unicode string in a place that only accepts ascii (like the console or as a path) you must tell Python that you want it to replace the non ascii characters best effort.

>> problem_str = u'This is not all ascii\xf8 man'
>> safe_str = problem_str.encode('ascii', 'ignore')
>> safe_str
'This is not all ascii man'

Encoding issues are prevented in the admin by the cautious handing of Django templating, but if you have ever added custom columns and forgotten to convert the values to ascii, or you override the str method for a model and forget to do this, you will get the same error, preventing template rendering.

If this string were saved into your (hopefully utf8) database there would be no problem, it looks like you are trying to upload a file that uses the title of an entity that has a non ascii character.

Answer 3

Hope this would help. In my case, I'm running django through daemontools.

Setting

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

in run script before executing manage.py resolved the issue with uploads filename

Answer 4

After investigating this some more I found out that I hadn't set the charset in my main Nginx config file:

http {
  charset  utf-8;
}

By adding the above, the problem disappeared and I think that this is the correct way of handling this issue.

Answer 5

akaihola's answer was helpful. For those who run django app with uWSGI managed via upstart script, just add these lines to your /etc/init/yourapp.conf

env LANG="en_US.utf8"
env LC_ALL="en_US.UTF-8"
env LC_LANG="en_US.UTF-8"

It solved the problem for me.

Answer 6

As said before, it is related to locale. For exemple, if you use gunicorn to serve your django application, you may have an init.d script (or, as me, a runit script), where you can set the locale.

To solve UnicodeEncodeError with file upload, put something like export LC_ALL=en_US.UTF8 in your script that run your app.

For example, this is mine (using gunicorn and runit):

#!/bin/bash
export LC_ALL=en_US.UTF8
cd /path/to/app/projectname
exec gunicorn_django -b localhost:8000 --workers=2

Also, you can check your locale in your template, using this in your view:

import locale
data_to_tpl = {'loc': locale.getlocale(), 'lod_def': locale.getdefaultlocale()}

And just disply {{loc}} - {{loc_def}} in your template.

You will have more information about your locale settings! That was very usefull for me.

Answer 7

It's hard to say without seeing a little more code but it looks to be related to this question: UnicodeDecodeError on attempt to save file through django default filebased backend.

Looking through the Django ticket mentioned it would seem you should follow something similar to the deployment docs on "If you get a UnicodeEncodeError":
https://docs.djangoproject.com/en/1.4/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror

(I know this is for Apache/mod_python but my guess would be it's the same root issue of file system encoding not being UTF-8 and there is a similar fix when using nginx)

EDIT: From what I can tell this nginx module would be the equivalent fix: http://wiki.nginx.org/NginxHttpCharsetModule

Answer 8

Another useful option that avoids rewriting code is to change the default encoding for python.

If you're using virtualenv you can change (or create if doesn't exist) env/lib/python2.7/sitecustomize.py and add:

import sys
sys.setdefaultencoding('utf-8')

or, if you are in a production system, you can do the same to /usr/lib/python2.7/sitecustomize.py

Answer 9

If you're using django and python 2.7 this fixes it for me:

@python_2_unicode_compatible
class Utente(models.Model):

see https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.encoding.python_2_unicode_compatible

Answer 10

Using python 2.7.8 and Django 1.7, I solved my problem by importing:

from __future__ import unicode_literals

and using force_text():

from django.utils.encoding import force_text

Answer 11

Just building on answers from this thread and others...

I had the same issue with genericpath.py giving a UnicodeEncodeError when attempting to upload a file name with non ASCII characters.

I was using nginx, uwsgi and django with python 2.7.

Everything was working OK locally but not on the server

Here are the steps I took 1. added to /etc/nginx/nginx.conf (did not fix the problem)

http {
    charset utf-8;
}

2. I added this line to etc/default/locale (did not fix the problem)

   LANGUAGE="en_US.UTF-8"

3. I followed the instructions here listed under the heading 'Success' https://code.djangoproject.com/wiki/ExpectedTestFailures (did not fix the problem)

   aptitude install language-pack-en-base
   

4. Found across this ticket https://code.djangoproject.com/ticket/17816 which suggested testing a view on the server to what was happening with locale information

In your view

import locale
locales = "Current locale: %s %s -- Default locale: %s %s" % (locale.getlocale() + locale.getdefaultlocale())

In your template

{{ locales }}

For me, the issue was that I had no locale and no default locale on my Ubuntu server (though I did have them on my local OSX dev machine) then files with non ASCII file names/paths will not upload correctly with python raising a UnicodeEncodeError, but only on the production server.

Solution

I added this to both my site and my site admin uwsgi config files e.g. /etc/uwsgi-emperor/vassals/my-site-config-ini file

env = LANG=en_US.utf8

Answer 12

None of the answers worked for me (using Apache on Ubuntu with Django 1.10); I chose to remove accents from the file name (normalize) as below:

def remove_accents(value):
    nkfd_form = unicodedata.normalize('NFKD', str(value))
    return "".join([c for c in nkfd_form if not unicodedata.combining(c)])

uploaded_file = self.cleaned_data['data']

# We need to remove accents to get rid of "UnicodeEncodeError: 'ascii' codec can't encode character" on Ubuntu
uploaded_file.name = remove_accents(uploaded_file.name)