[英]PySpark DataFrame: Change cell value based on min/max condition in another column
我有以下pySpark數據幀:
+------------------+------------------+--------------------+--------------+-------+
| col1| col2| col3| X| Y|
+------------------+------------------+--------------------+--------------+-------+
|2.1729247374294496| 3.558069532647046| 6.607603368496324| 1| null|
|0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| null|
|0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| null|
| 2.608497168338446| 3.529397129549324| 0.373034222141551| 2| null|
+------------------+------------------+--------------------+--------------+-------+
這是一個相當簡單的操作,我可以很容易地用熊貓做。 但是,我需要只使用pySpark。
我想做以下(我會寫一些偽代碼):
在col3 == max(col3)的行中,將Y從null更改為'K'
在剩下的行中,在col1 == max(col1)的行中,將Y從null更改為'Z'
在其余行中,在col1 == min(col1)的行中,將Y從null更改為“U”
在剩下的行中:將Y從null更改為“I”。
因此,預期的輸出是:
+------------------+------------------+--------------------+--------------+-------+
| col1| col2| col3| X| Y|
+------------------+------------------+--------------------+--------------+-------+
|2.1729247374294496| 3.558069532647046| 6.607603368496324| 1| K|
|0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| U|
|0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| I|
| 2.608497168338446| 3.529397129549324| 0.373034222141551| 2| Z|
+------------------+------------------+--------------------+--------------+-------+
完成后,我需要使用此表作為另一個表的查找:
+--------------------+--------+-----+------------------+--------------+------------+
| x1| x2| x3| x4| X| d|
+--------------------+--------+-----+------------------+--------------+------------+
|0057f68a-6330-42a...| 2876| 30| 5.989999771118164| 0| 20171219|
|05cc0191-4ee4-412...| 108381| 34|24.979999542236328| 3| 20171219|
|06f353af-e9d3-4d0...| 118798| 34| 0.0| 3| 20171219|
|0c69b607-112b-4f3...| 20993| 34| 0.0| 0| 20171219|
|0d1b52ba-1502-4ff...| 23817| 34| 0.0| 0| 20171219|
我想使用第一個表作為查找在第二個表中創建一個新列。 應使用第二個表中的X列作為鍵在第一個表的第Y列中查找新列的值(因此我們在第一個表中的列Y中查找與第X列中的值對應的值,這些值來自第X列中的值第二表)。
UPD:我需要一個滿足兩個條件的穩健解決方案,例如:
+------------------+------------------+--------------------+--------------+-------+
| col1| col2| col3| X| Y|
+------------------+------------------+--------------------+--------------+-------+
| 2.608497168338446| 3.558069532647046| 6.607603368496324| 1| null|
|0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| null|
|0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| null|
|2.1729247374294496| 3.529397129549324| 0.373034222141551| 2| null|
+------------------+------------------+--------------------+--------------+-------+
在這種情況下,行0滿足max('col3')和max('col1')條件。
所以需要發生的是:
第0行變為'K'
第3行變為'Z'(因為剩下的行(0已經'K'第3行滿足max('col1')條件
第1行變為'U'
第2行變為“我”
我不能在表1中有多行,其中包含'I'。
計算聚合:
from pyspark.sql import functions as F
df = spark.createDataFrame([
(2.1729247374294496, 3.558069532647046, 6.607603368496324, 1),
(0.2654841575294071, 1.2633077949463256, 0.023578679968183733, 0),
(0.4253301781296708, 3.4566490739823483, 0.11711202266039554, 3),
(2.608497168338446, 3.529397129549324, 0.373034222141551, 2)
], ("col1", "col2", "col3", "x"))
min1, max1, max3 = df.select(F.min("col1"), F.max("col1"), F.max("col3")).first()
添加列when
:
y = (F.when(F.col("col3") == max3, "K") # In row where col3 == max(col3), change Y from null to 'K'
.when(F.col("col1") == max1, "Z") # In the remaining rows, in the row where col1 == max(col1), change Y from null to 'Z'
.when(F.col("col1") == min1, "U") # In the remaining rows, in the row where col1 == min(col1), change Y from null to 'U'
.otherwise("I")) # In the remaining row: change Y from null to 'I'
df_with_y = df.withColumn("y", y)
df_with_y.show()
# +------------------+------------------+--------------------+---+---+
# | col1| col2| col3| x| y|
# +------------------+------------------+--------------------+---+---+
# |2.1729247374294496| 3.558069532647046| 6.607603368496324| 1| K|
# |0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| U|
# |0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| I|
# | 2.608497168338446| 3.529397129549324| 0.373034222141551| 2| Z|
# +------------------+------------------+--------------------+---+---+
應使用第二個表中的X列作為鍵,在第一個表的第Y列中查找新列的值
df_with_y.select("x", "Y").join(df2, ["x"])
如果y
已經存在,並且您要保留非空值:
df_ = spark.createDataFrame([
(2.1729247374294496, 3.558069532647046, 6.607603368496324, 1, "G"),
(0.2654841575294071, 1.2633077949463256, 0.023578679968183733, 0, None),
(0.4253301781296708, 3.4566490739823483, 0.11711202266039554, 3, None),
(2.608497168338446, 3.529397129549324, 0.373034222141551, 2, None)
], ("col1", "col2", "col3", "x", "y"))
min1_, max1_, max3_ = df.filter(F.col("y").isNull()).select(F.min("col1"), F.max("col1"), F.max("col3")).first()
y_ = (F.when(F.col("col3") == max3_, "K")
.when(F.col("col1") == max1_, "Z")
.when(F.col("col1") == min1_, "U")
.otherwise("I"))
df_.withColumn("y", F.coalesce(F.col("y"), y_)).show()
# +------------------+------------------+--------------------+---+---+
# | col1| col2| col3| x| y|
# +------------------+------------------+--------------------+---+---+
# |2.1729247374294496| 3.558069532647046| 6.607603368496324| 1| G|
# |0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| U|
# |0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| I|
# | 2.608497168338446| 3.529397129549324| 0.373034222141551| 2| K|
# +------------------+------------------+--------------------+---+---+
如果遇到數值精度問題,可以嘗試:
threshold = 0.0000001 # Choose appropriate
y_t = (F.when(F.abs(F.col("col3") - max3) < threshold, "K") # In row where col3 == max(col3), change Y from null to 'K'
.when(F.abs(F.col("col1") - max1) < threshold, "Z") # In the remaining rows, in the row where col1 == max(col1), change Y from null to 'Z'
.when(F.abs(F.col("col1") - min1) < threshold, "U") # In the remaining rows, in the row where col1 == min(col1), change Y from null to 'U'
.otherwise("I")) # In the remaining row: change Y from null to 'I'
df.withColumn("y", y_t).show()
# +------------------+------------------+--------------------+---+---+
# | col1| col2| col3| x| y|
# +------------------+------------------+--------------------+---+---+
# |2.1729247374294496| 3.558069532647046| 6.607603368496324| 1| K|
# |0.2654841575294071|1.2633077949463256|0.023578679968183733| 0| U|
# |0.4253301781296708|3.4566490739823483| 0.11711202266039554| 3| I|
# | 2.608497168338446| 3.529397129549324| 0.373034222141551| 2| Z|
# +------------------+------------------+--------------------+---+---+
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.