[英]Mode of row as a new column in PySpark DataFrame
Is it possible to add a new column based on the maximum of previous columns where the previous columns are string literals. 是否可以根据先前列的最大值添加新列,其中先前列是字符串文字。 Consider following dataframe:
考虑以下数据框:
df = spark.createDataFrame(
[
('1',25000,"black","black","white"),
('2',16000,"red","black","white"),
],
['ID','cash','colour_body','colour_head','colour_foot']
)
Then the target frame should look like this: 然后目标框架应如下所示:
df = spark.createDataFrame(
[
('1',25000,"black","black","white", "black" ),
('2',16000,"red","black","white", "white" ),
],
['ID','cash','colour_body','colour_head','colour_foot', 'max_v']
)
If there is no maximum detectable, then the last valid colour should be used. 如果没有最大可检测值,则应使用最后一个有效颜色。
Is there some kind of counter possibility available or udf? 是否存在某种反可能性或udf?
Define a UDF around statistics.mode
to compute the row-wise mode with the required semantics: 在
statistics.mode
周围定义一个UDF,以使用所需的语义计算逐行模式:
import statistics
from pyspark.sql.functions import udf, col
from pyspark.sql.types import StringType
def mode(*x):
try:
return statistics.mode(x)
except statistics.StatisticsError:
return x[-1]
mode = udf(mode, StringType())
df.withColumn("max_v", mode(*[col(c) for c in df.columns if 'colour' in c])).show()
+---+-----+-----------+-----------+-----------+-----+
| ID| cash|colour_body|colour_head|colour_foot|max_v|
+---+-----+-----------+-----------+-----------+-----+
| 1|25000| black| black| white|black|
| 2|16000| red| black| white|white|
+---+-----+-----------+-----------+-----------+-----+
For the general case of any number of columns, the udf
solution by @cs95 is the way to go. 对于任何列数一般情况下,
udf
通过@ cs95的解决方案是要走的路。
However, in this specific case where you have only 3 columns you can actually simplify the logic using just pyspark.sql.functions.when
, which will be more efficient than using a udf
. 然而,在这种特定的情况下,你只有3列你其实可以简化仅使用逻辑
pyspark.sql.functions.when
,这将是比使用更高效的udf
。
from pyspark.sql.functions import col, when
def mode_of_3_cols(body, head, foot):
return(
when(
(body == head)|(body == foot),
body
).when(
(head == foot),
head
).otherwise(foot)
)
df.withColumn(
"max_v",
mode_of_3_cols(col("colour_body"), col("colour_head"), col("colour_foot"))
).show()
#+---+-----+-----------+-----------+-----------+-----+
#| ID| cash|colour_body|colour_head|colour_foot|max_v|
#+---+-----+-----------+-----------+-----------+-----+
#| 1|25000| black| black| white|black|
#| 2|16000| red| black| white|white|
#+---+-----+-----------+-----------+-----------+-----+
You just need to check if any two columns are equal- if yes, then that value has to be the mode. 您只需要检查两列是否相等-如果是,则该值必须为mode。 If not, return the last column.
如果不是,则返回最后一列。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.