![](/img/trans.png)
[英]How to populate a Spark DataFrame column based on another column's value?
[英]Update a column value in a spark dataframe based another column
我有一个火花 dataframe,如下所述。
val data = spark.sparkContext.parallelize(Seq(
(1,"", "SNACKS", "BISCUITS - AMBIENT", "BISCUITS - AMBIENT", "", "REFLETS DE FRANCE CROQUANT", "UNCOATED BISCUIT", "NO PROMOTION", "", "", "400G","",""),
(2,"GROCERY", "BISCUITS", "SWEET BISCUITS ", "BISCUITS - AMBIENT", "", "", "AMBIENT BISCUIT", "NO PROMOTION", "", "", "400G","","CHOCOS")
))
.toDF("id", "c4", "c1001", "c1002", "c1003", "c1008", "c1008_unmasked", "c1009", "c1011", "c1012", "c1013", "c1015", "c1016", "c1016_unmasked")
data.show(false)
样本输入:
+---+-------+--------+------------------+------------------+-----+--------------------------+-----------------+------------+-----+-----+-----+-----+--------------+
|id |c4 |c1001 |c1002 |c1003 |c1008|c1008_unmasked |c1009 |c1011 |c1012|c1013|c1015|c1016|c1016_unmasked|
+---+-------+--------+------------------+------------------+-----+--------------------------+-----------------+------------+-----+-----+-----+-----+--------------+
|1 | |SNACKS |BISCUITS - AMBIENT|BISCUITS - AMBIENT| |REFLETS DE FRANCE CROQUANT|UNCOATED BISCUIT|NO PROMOTION| | |400G | | |
|2 |GROCERY|BISCUITS|SWEET BISCUITS |BISCUITS - AMBIENT| | |AMBIENT BISCUIT |NO PROMOTION| | |400G | |CHOCOS |
+---+-------+--------+------------------+------------------+-----+--------------------------+-----------------+------------+-----+-----+-----+-----+--------------+
仅当相同的cXXXX_unmasked具有值时,才需要使用值“MASKED”填充列cXXXX 。 请检查示例 output 以获得更好的理解。
+---+-------+--------+------------------+------------------+------+--------------------------+-----------------+------------+-----+-----+-----+------+--------------+
|id |c4 |c1001 |c1002 |c1003 |c1008 |c1008_unmasked |c1009 |c1011 |c1012|c1013|c1015|c1016 |c1016_unmasked|
+---+-------+--------+------------------+------------------+------+--------------------------+-----------------+------------+-----+-----+-----+------+--------------+
|1 | |SNACKS |BISCUITS - AMBIENT|BISCUITS - AMBIENT|MASKED|REFLETS DE FRANCE CROQUANT|UNCOATED BISCUIT|NO PROMOTION| | |400G | | |
|2 |GROCERY|BISCUITS|SWEET BISCUITS |BISCUITS - AMBIENT| | |AMBIENT BISCUIT |NO PROMOTION| | |400G |MASKED|CHOCOS |
+---+-------+--------+------------------+------------------+------+--------------------------+-----------------+------------+-----+-----+-----+------+--------------+
提前致谢
这是我的尝试。
val cols = data.columns.filter(_.endsWith("_unmasked"))
val new_data = cols.foldLeft(data) { (df, c) =>
df.withColumn(c.split("_").head, when(col(c) =!= "" && col(c).isNotNull, lit("MASKED")).otherwise(col(c)))
}
new_data.show
+---+-------+--------+------------------+------------------+------+--------------------+-----------------+------------+-----+-----+-----+------+--------------+
| id| c4| c1001| c1002| c1003| c1008| c1008_unmasked| c1009| c1011|c1012|c1013|c1015| c1016|c1016_unmasked|
+---+-------+--------+------------------+------------------+------+--------------------+-----------------+------------+-----+-----+-----+------+--------------+
| 1| | SNACKS|BISCUITS - AMBIENT|BISCUITS - AMBIENT|MASKED|REFLETS DE FRANCE...|UNCOATED BISCUIT|NO PROMOTION| | | 400G| | |
| 2|GROCERY|BISCUITS| SWEET BISCUITS |BISCUITS - AMBIENT| | | AMBIENT BISCUIT|NO PROMOTION| | | 400G|MASKED| CHOCOS|
+---+-------+--------+------------------+------------------+------+--------------------+-----------------+------------+-----+-----+-----+------+--------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.