如何在python apache beam中的窗口中订购元素？

Question

I noticed that java apache beam has class groupby.sortbytimestamp does python have that feature implemented yet? If not what would be the way to sort elements in a window? I figure I could sort the entire window in a DoFn, but I would like to know if there is a better way.

Answer 1

There is not currently built-in value sorting in Beam (in either Python or Java). Right now, the best option is to sort the values yourself in a DoFn like you mentioned.

Answer 2

Here's a solution using a CombineFn. It has the added bonus of deduplicating data using the TreeSet. You also should make sure your data for a window is small enough to fit in memory on a single worker.

public static class DedupAndSortByTime extends Combine.CombineFn<MarketData, TreeSet<MarketData>, List<MarketData>> {
@Override
public TreeSet<MarketData> createAccumulator() {
    return new TreeSet<>(Comparator
            .comparingLong(MarketData::getEventTime)
            .thenComparing(MarketData::getOrderbookType));
}

@Override
public TreeSet<MarketData> addInput(TreeSet<MarketData> accum, MarketData input) {
    accum.add(input);
    return accum;
}

@Override
public TreeSet<MarketData> mergeAccumulators(Iterable<TreeSet<MarketData>> accums) {

    TreeSet<MarketData> merged = createAccumulator();
    for (TreeSet<MarketData> accum : accums) {
        merged.addAll(accum);
    }
    return merged;
}

@Override
public List<MarketData> extractOutput(TreeSet<MarketData> accum) {
    return Lists.newArrayList(accum.iterator());
}

}