UnicodeEncodeError when uploading files in Django using Supervisor and Gunicorn

July, 21 2012

Issues related uploading files with unicode filenames have long history with Django and Linux.

There are many stack overflow questions and blog posts on the subject already, but very few deal with solving the problem when using Supervisord and Gunicorn directly.

If you are using apache2 and mod_python you can configure your as per the docs:

https://code.djangoproject.com/wiki/django_apache_and_mod_wsgi#AdditionalTweaking

Fundamentally all fixes have the same goal: configure the environment of the server running Django to use the correct locale, so sys.getfilesystemencoding() will return 'UTF-8' instead of 'ANSI_X3.4-1968'. What makes solving this problem especially hard is there are many possible solutions and picking the right one for you situation can be difficult.

Ignore Nginx Related Advice

Since you are using Gunicorn, I am going to guess you are using Nginx and you will come across posts saying you need to add charset utf-8 to the http section in the root Nginx config file. Though doing this is probably generally a good idea, it will not solve the problem. Nginx knows nothing about Python, Django or Gunicorn, except that when it makes requests down a unix or TCP socket it gets back bytes. No setting in Nginx will affect the environment Python is running in.

Environment and Supervisord

The most helpful advice is to add a line similar to this:

environment=LANG=en_CA.UTF-8,LC_ALL=en_CA.UTF-8,LC_LANG=en_CA.UTF-8

To your supervisord.conf file in the [supervisord] section. Your locale maybe different.

This is good advice, but make sure that supervisor has actually read and is using this configuration directive is difficult. Supervisor tries hard not to completely restart since that means all of its processes/services will be forced to restart, but this is one situation where as far as I can tell it is necessary.

Unfortunately you cannot just do /etc/init.d/supervisord restart, since this doesn't really restart anything. You must issue a stop first, then issue a start.

You could potentially add the environment line to a specific supervisord .conf file, but that isn't supported in all versions.

You can use the following test supervisor program to help debug if your locale settings are working or not:

[program:locale]
command=python -c "import sys; print sys.getfilesystemencoding();"
directory=/
user=www-data
autostart=true
autorestart=true
redirect_stderr=True

You can adjust the settings to suit your purpose.

Also see this very handy blog post on the matter: http://tech.barszcz.info/2012/05/17/django-unicodeencodeerror-when-uploading-files/.

If you discover more about how to solve this problem please contact me and I will update this post.


Tweet comments, corrections, or high fives to @amjoconn