pysqlite2: ProgrammingError - You must not use 8-bit bytestrings

Question

I\'m currently persisting filenames in a sqlite database for my own purposes. Whenever I try to insert a file that has a special character (like é etc.), it throws the follo

Answer 1

You need to specify the encoding of filename for conversion to Unicode, for example: filename.decode('utf-8'). Just using unicode(...) picks the console encoding, which is often unreliable (and often ascii).

Answer 2

Try to change to this:

cursor.execute("select * from musiclibrary where absolutepath = ?;",
    [unicode(filename,'utf8')])

In your filename origin not encode with utf8, change utf8 to your encoding.

Answer 3

Have you tried to pass the unicode string directly:

cursor.execute("select * from musiclibrary where absolutepath = ?;",(u'namé',))

You will need to add the file encoding at the beginning of the script:

# coding: utf-8

Answer 4

You figured this out already, but:

I don't think you could actually get that ProgrammingError exception from cursor.execute("select * from musiclibrary where absolutepath = ?;", [filename.decode("utf-8")]), as the question currently states.

Either the utf-8 decode would explode, or the cursor.execute call would be happy with the result.

Answer 5

You should pass as Unicode the arguments of your SQL statement.

Now, it all depends on how you obtain the filename list. Perhaps you're reading the filesystem using os.listdir or os.walk? If that is the case, there is a way to have directly the filenames as Unicode just by passing a Unicode argument to either of these functions:
Examples:

• os.listdir(u'.')
• os.walk(u'.')

Of course, you can substitute the u'.' directory with the actual directory whose contents you are reading. Just make sure it's a Unicode string.