How to resolve “iterator should return strings, not bytes”

Question

I am trying to import a CSV file, using a form to upload the file from the client system. After I have the file, I\'ll take parts of it and populate a model in my app. However,

Answer 1

request.FILES gives you binary files, but the csv module wants to have text-mode files instead.

You need to wrap the file in a io.TextIOWrapper() instance, and you need to figure out the encoding:

from io import TextIOWrapper

f = TextIOWrapper(request.FILES['filename'].file, encoding=request.encoding)

It'd probably be better if you took the charset parameter from the Content-Type header if provided; that is what the client tells you the character set is.

You cannot work around needing to know the correct encoding for the file data; you can force interpretation as ASCII, for example, by providing a errors keyword as well (setting it to 'replace' or 'ignore'), but that does lead to data loss:

f = TextIOWrapper(request.FILES['filename'].file, encoding='ascii', errors='replace')

Using TextIOWrapper will only work when using Django 1.11 and later (as this changeset added the required support). In earlier versions, you can monkey-patch the support in after the fact:

from django.core.files.utils import FileProxyMixin

if not hasattr(FileProxyMixin, 'readable'):
    # Pre-Django 1.11, add io.IOBase support, see
    # https://github.com/django/django/commit/4f474607de9b470f977a734bdd47590ab202e778        
    def readable(self):
        if self.closed:
            return False
        if hasattr(self.file, 'readable'):
            return self.file.readable()
        return True

    def writable(self):
        if self.closed:
            return False
        if hasattr(self.file, 'writable'):
            return self.file.writable()
        return 'w' in getattr(self.file, 'mode', '')

    def seekable(self):
        if self.closed:
            return False
        if hasattr(self.file, 'seekable'):
            return self.file.seekable()
        return True

    FileProxyMixin.closed = property(
        lambda self: not self.file or self.file.closed)
    FileProxyMixin.readable = readable
    FileProxyMixin.writable = writable
    FileProxyMixin.seekable = seekable

Answer 2

In python 3, I used:

import csv
from io import StringIO
csvf = StringIO(xls_file.read().decode())
reader = csv.reader(csvf, delimiter=',')

xls_file being the file got from the POST form. I hope it helps.

Answer 3

Fuse your two methods, this never fails in Python 3.5.2 and Django 1.9

delimitador = list_delimitadores[int(request.POST['delimitador'])][1]
try:
    text = TextIOWrapper(request.FILES['csv_x'].file, encoding='utf-8 ', errors='replace')
    reader = csv.reader(text, delimiter=delimitador)
except:
    text = StringIO(request.FILES['csv_x'].file.read().decode())
    reader = csv.reader(text, delimiter=delimitador)