python3 UnicodeEncodeError: 'charmap' codec can't encode characters in position 95-98: character maps to

Question

A month ago I encountered this Github: https://github.com/taraslayshchuk/es2csv

I installed this package via pip3 in Linux ubuntu. When I wanted to use this package,

Answer 1

It's due to the default behaviour of open in Python 3. By default, Python 3 will open files in Text mode, which means that it also has to apply a text decoding, such as utf-8 or ASCII, for every character it reads.

Python will use your locale to determine the most suitable encoding. On OS X and Linux, this is usually UTF-8. On Windows, it'll use an 8-bit character set, such windows-1252, to match the behaviour of Notepad.

As an 8-bit character set only has a limited number of characters, it's very easy to end up trying to write a character not supported by the character set. For example, if you tried to write a Hebrew character with Windows-1252, the Western European character set.

To resolve your problem, you simply need to override the automatic encoding selection in open and hardcode it to use UTF-8:

for line in open(self.tmp_file, 'r', encoding='utf-8'):