如何使用 Apache Beam 动态地将文件写入谷歌存储桶？

Question

I'm trying to write a few files to google bucket using Apache Beam, but the file location and folder names are generated from first index in the file, so how do I create this dictionary and write my files to it?

metadata = (data_from_test |'CSVConversionMeta' >> beam.ParDo(WriteToCSVmeta())|'Writing To File' >> beam.io.WriteToText('gs://tester1212/CIK/YEAR/FILING/metadata.csv'))

So, this is the code where I write the file, but I want the YEAR to get fetched from the csv and create a folder during runtime.

Answer 1

If your date is known before the Pipeline start, you can do this

year="2020"
metadata = (data_from_test |'CSVConversionMeta' >> beam.ParDo(WriteToCSVmeta())|'Writing To File' >> beam.io.WriteToText('gs://tester1212/CIK/{}/FILING/metadata.csv'.format(year)))