Python: Traceback codecs.charmap_decode(input,self.errors,decoding_table)[0]

Question

Following is sample code, aim is just to merges text files from give folder and it\'s sub folder. i am getting Traceback occasionally so not sure where to look. also need so

Answer 1

The error is thrown because Python 3 opens your files with a default encoding that doesn't match the contents.

If all you are doing is copying file contents, you'd be better off using the shutil.copyfileobj() function together with opening the files in binary mode. That way you avoid encoding issues altogether (as long as all your source files are the same encoding of course, so you don't end up with a target file with mixed encodings):

import shutil
import os.path

with open('C:\\Dropbox\\Python\\master.txt','wb') as output:
    for path, f_name in files:
        with open(os.path.join(path, f_name), 'rb') as input:
            shutil.copyfileobj(input, output)
        output.write(b'\n') # insert extra newline between files

I've cleaned up the code a little to use context managers (so your files get closed automatically when done) and to use os.path to create the full path for your files.

If you do need to process your input line by line you'll need to tell Python what encoding to expect, so it can decode the file contents to python string objects:

open(path, mode, encoding='UTF8')

Note that this requires you to know up front what encoding the files use.

Read up on the Python Unicode HOWTO if you have further questions about python 3, files and encodings.