sys.argv as bytes in Python 3k

Question

As Python 3k introduces strict distinction between strings and bytes, command line arguments in the array sys.argv are presented as strings. Sometimes it is necessary to tre

Answer 1

Note that the error is a UnicodeEncodeError rather than a UnicodeDecodeError. Python is preserving the exact bytes passed on the command line (via the PEP 383 surrogateescape error handler), but those bytes are not valid UTF-8 and hence can't be encoded as such for writing to the console.

The best way to deal with this is to use the application level knowledge of the correct encoding to reinterpret the command line argument inside the application, as in the following example code:

$ python3.2 -c "import os, sys; print(os.fsencode(sys.argv[1]).decode('latin-1'))" `echo français|iconv -t latin1`
français

The os.fsencode function invocation reverses the transformation Python applied automatically when processing the command line arguments. The decode('latin-1') method invocation then performs the correct conversion in order to get a properly decoded string.

Python 3.2 added os.fsencode to specifically to make this kind of problem easier to deal with.

For Python 3.1, the equivalent construct for os.fsencode(sys.argv[1]) is sys.argv[1].encode(sys.getfilesystemencoding(), 'surrogateescape')

Edit Feb 2013: updated for Python 3.2+, and to avoid assuming that Python autodetected "UTF-8" as the command line encoding

Answer 2

You can do:

sys.argv[1].encode() or, if you know the encoding use it as argument or call bytes(sys.argv[1], 'latin-1').

Both should give you a byte representation of the unicode string.

By default, Python3 uses UTF-8.