如何从 s3 将镶木地板文件读入 PCollection？

Question

My problem is simple: I want to read a parquet file from s3 into a PCollection in Apache Beam using the Python Sdk.

I know of the apache_beam.io.parquetio module but this one does not seem to be able to read from s3 directly (or does it?).

I know of the apache_beam.io.aws.s3io module but this one seems to return an s3 file object or something that is not a PCollection anyway (or does it?).

So what's the best way to do this?

Answer 1

if you install beam with the aws requirement

pip install 'apache-beam[aws]'

You can just pass in an s3 filename to read from it

filename = "s3://bucket-name/...
beam.io.ReadFromParquet(filenam)