S3 文件到本地使用 luigi 引发 UnicodeDecodeError

Question

I am copying a pdf file to local, using the following piece of code:

with self.input_target().open('r') as r:
    with self.output_target().open('w') as w:
       for line in r: 
           w.write(line) 

Which is based in this question (kind of)

But when I execute that code I get the following:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11:
      invalid continuation byte

I tried this other approach, without good results:

   with self.input_target().open('r') as r, self.output_target().open('w') as w:                                                                                                                        
       w.write(r.read())                         

What is the correct way of doing it?

Answer 1

It seems that you're dealing with a binary file as if it was text - but it's not. You probably need to do something like this:

with self.input().open('r') as i, self.output().open('w') as o:
    o.write(i.read())

(not tested!)

Also, I think you may find this answer useful: Python writing binary files, bytes

Answer 2

I ended using luigi.s3.S3Client and the method get which copies a binary file from/to the s3 .

Snippet of code:

import luigi

class ATask(luigi.Task):
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()
    ...

    def run(self):
        client = luigi.s3.S3Client()
        client.get(s3_path, local_path)  ## This gets the file
        ...

I think that the underlying reason is that luigi uses boto for getting/putting files from/to the s3 . (As you can see in the source code)

Answer 3

Another option would be to use the format kwarg luigi.format.Nop to Target to indicate that it is a binary file that is being read.

eg

def output(self):
    return LocalTarget(target, format=luigi.format.Nop)