UnicodeDecodeError when reading CSV file in Pandas with Python

后端 未结 21 2214
野趣味
野趣味 2020-11-22 04:27

I\'m running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error...

File "C:\\Importer\\src         


        
相关标签:
21条回答
  • 2020-11-22 05:10

    Check the encoding before you pass to pandas. It will slow you down, but...

    with open(path, 'r') as f:
        encoding = f.encoding 
    
    df = pd.read_csv(path,sep=sep, encoding=encoding)
    

    In python 3.7

    0 讨论(0)
  • 2020-11-22 05:12

    read_csv takes an encoding option to deal with files in different formats. I mostly use read_csv('file', encoding = "ISO-8859-1"), or alternatively encoding = "utf-8" for reading, and generally utf-8 for to_csv.

    You can also use one of several alias options like 'latin' instead of 'ISO-8859-1' (see python docs, also for numerous other encodings you may encounter).

    See relevant Pandas documentation, python docs examples on csv files, and plenty of related questions here on SO. A good background resource is What every developer should know about unicode and character sets.

    To detect the encoding (assuming the file contains non-ascii characters), you can use enca (see man page) or file -i (linux) or file -I (osx) (see man page).

    0 讨论(0)
  • 2020-11-22 05:12

    In my case, a file has USC-2 LE BOM encoding, according to Notepad++. It is encoding="utf_16_le" for python.

    Hope, it helps to find an answer a bit faster for someone.

    0 讨论(0)
提交回复
热议问题