Pyspark 加入数据框

Question

我有两个火花数据框。

df1

id    product  price
0     x        100
1     y        120
2     z        110
3     x        150
4     x        100

和 df2

id    unique_products 
0     x        
1     y        
2     z         

我怎样才能得到这个结果：

id    unique_products  prices
0     x                [100, 150, 100]                      
1     y                [120]
2     z                [110]

Answer 1

您可以按product分组并在price上应用collect_list 。 最后加入df2以获得id 。

from pyspark.sql import functions as F

data1 = [(0, "x", 100,),
        (1, "y", 120,),
        (2, "z", 110,),
        (3, "x", 150,),
        (4, "x", 100,), ]

data2 = [(0, "x", ), (1, "y", ), (2, "z", ), ]

df1 = spark.createDataFrame(data1,("id", "product", "price",)) 
df2 = spark.createDataFrame(data2,("id", "unique_products", ))

df_prices = df1.groupBy("product").agg(F.collect_list("price").alias("prices")).selectExpr("product as unique_products", "prices")

df2.join(df_prices, ["unique_products"]).select("id", "unique_products", "prices").show()

Output

+---+---------------+---------------+
| id|unique_products|         prices|
+---+---------------+---------------+
|  0|              x|[100, 150, 100]|
|  1|              y|          [120]|
|  2|              z|          [110]|
+---+---------------+---------------+