UnicodeEncodeError: 'ascii' codec can't encode character

Question

When uploading files with non-ASCII characters I get UnicodeEncodeError:

Exception Type: UnicodeEncodeError at /admin/studio/newsitem/add/
Exception Value: \         


        

Answer 1

For anyone encountering this problem when running Django with Supervisor, the solution is to add e.g. the following to the supervisord section of Supervisor's configuration:

environment=LANG="en_US.utf8", LC_ALL="en_US.UTF-8", LC_LANG="en_US.UTF-8"

This solved the problem for me in Supervisor 3.0a8 running on Debian Squeeze.

Also make sure Supervisor re-reads the configuration by running:

supervisorctl reread
supervisorctl restart myservice

(thanks @Udi)

---

For upstart, add in your /etc/init/myservice.conf:

env LANG="en_US.utf8"
env LC_ALL="en_US.UTF-8"
env LC_LANG="en_US.UTF-8"`

(thanks @Andrii Zarubin; see Environment Variables in Upstart documentation for more information)

Answer 2

None of the answers worked for me (using Apache on Ubuntu with Django 1.10); I chose to remove accents from the file name (normalize) as below:

def remove_accents(value):
    nkfd_form = unicodedata.normalize('NFKD', str(value))
    return "".join([c for c in nkfd_form if not unicodedata.combining(c)])

uploaded_file = self.cleaned_data['data']

# We need to remove accents to get rid of "UnicodeEncodeError: 'ascii' codec can't encode character" on Ubuntu
uploaded_file.name = remove_accents(uploaded_file.name)

Answer 3

Hope this would help. In my case, I'm running django through daemontools.

Setting

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

in run script before executing manage.py resolved the issue with uploads filename

Answer 4

If you're using django and python 2.7 this fixes it for me:

@python_2_unicode_compatible
class Utente(models.Model):

see https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.encoding.python_2_unicode_compatible

Answer 5

Just building on answers from this thread and others...

I had the same issue with genericpath.py giving a UnicodeEncodeError when attempting to upload a file name with non ASCII characters.

I was using nginx, uwsgi and django with python 2.7.

Everything was working OK locally but not on the server

Here are the steps I took 1. added to /etc/nginx/nginx.conf (did not fix the problem)

http {
    charset utf-8;
}

2. I added this line to etc/default/locale (did not fix the problem)

   LANGUAGE="en_US.UTF-8"

3. I followed the instructions here listed under the heading 'Success' https://code.djangoproject.com/wiki/ExpectedTestFailures (did not fix the problem)

   aptitude install language-pack-en-base
   

4. Found across this ticket https://code.djangoproject.com/ticket/17816 which suggested testing a view on the server to what was happening with locale information

In your view

import locale
locales = "Current locale: %s %s -- Default locale: %s %s" % (locale.getlocale() + locale.getdefaultlocale())

In your template

{{ locales }}

For me, the issue was that I had no locale and no default locale on my Ubuntu server (though I did have them on my local OSX dev machine) then files with non ASCII file names/paths will not upload correctly with python raising a UnicodeEncodeError, but only on the production server.

Solution

I added this to both my site and my site admin uwsgi config files e.g. /etc/uwsgi-emperor/vassals/my-site-config-ini file

env = LANG=en_US.utf8

Answer 6

After investigating this some more I found out that I hadn't set the charset in my main Nginx config file:

http {
  charset  utf-8;
}

By adding the above, the problem disappeared and I think that this is the correct way of handling this issue.