处理输入文件模式时如何在 apache 光束中的 DoFn 中获取文件名

Question

I am processing large no of files which are inside one directory. I want to add fileName in metadata of processed data output. So that if something goes wrong while processing, we can check that what is input file for the processed record.

Is there a way i can get file name inside my DoFn. I am using apache beam 2.19.0 version

Input file location - gs://bucket/extracted-files/*

Answer 1

You can use transforms available in FileIO class for this purpose.

Specially you can use FileIO.match() followed by FileIO.readMatches() which results in a PCollection of ReadableFile objects. For each ReadableFile you have access to a byte channel for reading as well as a Metadata object that contains the name of the file.