GCP Dataflow 抛出异常 Shuffle key too large

Question

I have a piece of code that groups my data but it throws an exception when I do output.

This class is used as key in KV

class CKey {
    private Long id;
    private Long subId;
}

It's part of my Dataflow job

TupleTag<CItem> itemsTuple = //...
TupleTag<CMeta> metaTuple = //...

//...

PCollection<KV<CKey, CItem>> items = null;
PCollection<KV<CKey, CMeta>> meta;

KeyedPCollectionTuple.of(itemsTuple, items).and(metaTuple, meta.next())
        .apply(CoGroupByKey.create())
        .apply(new CustomGroupPairsFn());

Custom function to join data

class CustomGroupPairsFn extends DoFn<KV<CKey, CoGbkResult>, MyCustomObject> {

        @ProcessElement
        public void processElement(@Element KV<CKey, CoGbkResult> element, OutputReceiver<MyCustomObject> out) {
            CoGbkResult pair = element.getValue();
            Iterator<CItem> citem = pair.getAll(ITEMS).iterator();
            Iterator<CMeta> cmeta = pair.getAll(METADATA).iterator();
            try {
                out.output(new MyCustomObject(citem.next(), cmeta));
            } catch (Exception e) {
                log.error("Error occurred", e);
            }
        }
    }

There is only 1 line of code in try and exception is thrown inside, exception:

How can I fix the issue?

Answer 1

This error happens because you're shuffling a key that is too large.

What does this mean? In Dataflow, the largest shuffle key allowed is 1.5 MB for streaming pipelines. You seem to have an element key that is larger than that.

Perhaps your pipeline has a GroupByKey/Shuffling operation somewhere unexpected, so that's why it would be great to have more details about it.