I\'m running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error...
File "C:\\Importer\\src
Check the encoding before you pass to pandas. It will slow you down, but...
with open(path, 'r') as f:
encoding = f.encoding
df = pd.read_csv(path,sep=sep, encoding=encoding)
In python 3.7
read_csv
takes an encoding
option to deal with files in different formats. I mostly use read_csv('file', encoding = "ISO-8859-1")
, or alternatively encoding = "utf-8"
for reading, and generally utf-8
for to_csv
.
You can also use one of several alias
options like 'latin'
instead of 'ISO-8859-1'
(see python docs, also for numerous other encodings you may encounter).
See relevant Pandas documentation, python docs examples on csv files, and plenty of related questions here on SO. A good background resource is What every developer should know about unicode and character sets.
To detect the encoding (assuming the file contains non-ascii characters), you can use enca
(see man page) or file -i
(linux) or file -I
(osx) (see man page).
In my case, a file has USC-2 LE BOM
encoding, according to Notepad++.
It is encoding="utf_16_le"
for python.
Hope, it helps to find an answer a bit faster for someone.