如何使用Apache Beam在Python中将有界pcollection转换为无界？

Question

I'm trying to transform a few Terabytes of mail logs stored in GCS without using too much memory.

As recommended in the guide , I'm adding timestamp to each element, split it to a sliding window and I have specified an (aggregation) trigger before sending it to a GroupByKey and a ParDo parser after. This should do, but still GroupByKey waits for all data to arrive. Why?

I have tried it with Direct and Google Dataflow runner as well.

What do I miss?

Here's the gist of the code:

    known_args, pipeline_options = parse_args(sys.argv)
    pipeline = beam.Pipeline(options=pipeline_options)
    lines = pipeline | 'read' >> beam.io.ReadFromText(known_args.input)
    parsed_json = lines \
      | 'cut' >> beam.Map(lambda x: x[39:])\
      | 'jsonify' >> beam.Map(json_loads).with_outputs('invalid_input', main='main')

    keyed_lines = parsed_json['main']\
      | 'key_filter' >> beam.Filter(lambda x: u'key' in x)\
      | 'keyer' >> beam.Map(lambda x: (x['key'], x))\
      | 'log_processed_rows' >> beam.Map(log_processed_rows)

    window_trigger = trigger.DefaultTrigger()

    windowed_lines = keyed_lines\
       | 'timestamp' >> beam.ParDo(AddTimestampDoFn())\
       | 'window' >> beam.WindowInto(beam.window.SlidingWindows(size=3600+600, period=3600), trigger=window_trigger, 
                                     accumulation_mode=trigger.AccumulationMode.DISCARDING) # 1 hour steps with 10 minutes of overlap

    results = windowed_lines \
        | 'groupbykey' >> beam.GroupByKey()\
        | 'parse' >> beam.ParDo(ParseSendingsDoFn()).with_outputs('too_few_rows', 'invalid_rows', 'missing_recipients', main='main_valid')  

    output = results['main_valid'] \
        | 'format' >> beam.Map(output_format)\
        | 'write' >> beam.io.WriteToText(known_args.output, file_name_suffix=".gz")

Answer 1

The default trigger requires the entire data set to be available before processing that is why it waits for all data to arrive.

Try using a Data-driven triggers or a Processing time triggers .

Furthermore, the SlidingWindows is mostly used for running averages and since you are only adding a timestamp the single global window might be a better option.

EDIT

Regarding the windowing, you can can have a streaming job that creates a pcollection from in-memory data and have a side input from the GCS storage.