S3 file to local using luigi raises UnicodeDecodeError

问题

I am copying a pdf file to local, using the following piece of code:

with self.input_target().open('r') as r:
    with self.output_target().open('w') as w:
       for line in r: 
           w.write(line)

Which is based in this question (kind of)

But when I execute that code I get the following:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11:
      invalid continuation byte

I tried this other approach, without good results:

   with self.input_target().open('r') as r, self.output_target().open('w') as w:                                                                                                                        
       w.write(r.read())

What is the correct way of doing it?

回答1:

It seems that you're dealing with a binary file as if it was text - but it's not. You probably need to do something like this:

with self.input().open('r') as i, self.output().open('w') as o:
    o.write(i.read())

(not tested!)

Also, I think you may find this answer useful: Python writing binary files, bytes

回答2:

I ended using luigi.s3.S3Clientand the method get which copies a binary file from/to the s3.

Snippet of code:

import luigi

class ATask(luigi.Task):
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()
    ...

    def run(self):
        client = luigi.s3.S3Client()
        client.get(s3_path, local_path)  ## This gets the file
        ...

I think that the underlying reason is that luigi uses boto for getting/putting files from/to the s3. (As you can see in the source code)

回答3:

Another option would be to use the format kwarg luigi.format.Nop to Target to indicate that it is a binary file that is being read.

e.g.

def output(self):
    return LocalTarget(target, format=luigi.format.Nop)

来源：https://stackoverflow.com/questions/33579409/s3-file-to-local-using-luigi-raises-unicodedecodeerror

标签

python

python-3.x

amazon-s3

luigi