apache_beam (python SDK) 是否支持.zip 压缩类型

Question

I am implementing a batch pipeline with apache beam that decompress json files, pre-process them and store them back on a given location within the filesystem.

Files could be compressed using ZIP or GZIP algorithms..

The decompression is working well with GZIP files but it fails on ZIP files... After investigating, I found that only GZIP, BZIP2 and DEFLATE compression types are only supported within the JAVA SDK but no python implementation exists.

Is there a work around to solve this without patching the apache beam Python SDK?

Answer 1

Beam Python does not support ZIP. There are two workarounds: you can read the files in a DoFn, or you can use the Java SDK's file IO via a Cross-Language Transform .

The read-via-dofn approach would look something like

filenames
  | beam.Map(lambda f: (f, None))
  | beam.GroupByKey() # The GroupByKey adds a fusion break so that files can be processed in parallel
  | beam.Map(lambda f: f[0])
  | beam.FlatMap(lambda f: [line for line in read(f)]