UTF-8 and os.listdir()

Question

I\'m having a bit of trouble with a file containing the \"ș\" character (that\'s \\xC8\\x99 in UTF-8 - LATIN SMALL LETTER S WITH COMMA BELOW).

I\'m crea

Answer 1

The OS X filesystem mostly uses decomposed characters rather than their combined form. You'll need to normalise the filenames back to the NFC combined normalised form:

import unicodedata
files = [unicodedata.normalize('NFC', f) for f in os.listdir(u'.')]

This processes filenames as unicode; you'd otherwise need to decode the bytestring to unicode first.

Also see the unicodedata.normalize() function documentation.