如何在apache beam中实现groupby(column1,column2)

Question

I need help in writing similar beam code in python for the following Spark sql code.

count_mnm_df = (mnm_df
     .select("State", "Color", "Count") 
     .groupBy("State", "Color") 
     .agg(count("Count").alias("Total")) 
     .orderBy("Total", ascending=False)

Answer 1

Probably the most straightforward mapping to above will be Beam SQL. See here for more information. Please see here for corresponding Python transform which also contains information regarding usage. Please note that support for Python SDK is achieved through Beam's cross-language transforms support which is relatively new.

You can also consider authoring a Beam pipeline using available Beam transforms that performs that same computation.

Note that Beam does not guarantee the order of elements of a PCollection .