![](/img/trans.png)
[英]How to populate a Spark DataFrame column based on another column's value?
[英]How to populate a new spark dataframe column with values from specific rows in another column. Need suggestions
我的问题是这样的:
I have a spark dataframe that looks like this
+-----------+---------------+
| id| name|
+-----------+---------------+
| 1| Total:|
| 2| Male:|
| 3| Under 5 years|
| 4| 5 to 9 years|
| 5| 10 to 14 years|
| 6| Female:|
| 7| Under 5 years|
| 8| 5 to 9 years|
| 9| 10 to 14 years|
+-----------+---------------+
I want to create a new DF with an added column that will look like this:
+-----------+---------------+---------------------+
| id| name| new_name|
+-----------+---------------+---------------------+
| 1| Total:| Total:|
| 2| Male:| Male:|
| 3| Under 5 years| Male: Under 5 years|
| 4| 5 to 9 years| Male: Under 5 years|
| 5| 10 to 14 years| Male: Under 5 years|
| 6| Female:| Female:|
| 7| Under 5 years|Female: Under 5 years|
| 8| 5 to 9 years|Female: Under 5 years|
| 9| 10 to 14 years|Female: Under 5 years|
+-----------+---------------+---------------------+
我没有任何值得展示的代码我正在寻找解决问题的方法。 我认为它会是这样的:
val dfB = dfA.withColum(row => aUDF(row))
我假设解决方案需要某种 UDF。 我假设它需要循环或映射并在任何时候在名称字段中找到带有“:”的行时更新“前缀”val。 但我不知道该怎么做。 任何想法将不胜感激。
Spark 2.4.3 你可以通过使用 split 和 last window 函数来实现这一点。
scala> import org.apache.spark.sql.expressions.Window
scala> var df = spark.createDataFrame(Seq((1,"Total:"), (2,"Male:"),(3, "Under 5 years"),(4,"5 to 9 years"),(5, "10 to 14 years"),(6,"Female:"),(7,"Under 5 years"),(8,"5 to 9 years"),(9, "10 to 14 years"))).toDF("id","name")
scala> df.show
+---+--------------+
| id| name|
+---+--------------+
| 1| Total:|
| 2| Male:|
| 3| Under 5 years|
| 4| 5 to 9 years|
| 5|10 to 14 years|
| 6| Female:|
| 7| Under 5 years|
| 8| 5 to 9 years|
| 9|10 to 14 years|
+---+--------------+
scala> var win =Window.orderBy(col("id"))
scala> var df2 =df.withColumn("name_1",last(when(split($"name",":")(1) ==="",$"name"),true).over(win))
scala> df2.withColumn("name",when($"name"===$"name_1",$"name").otherwise(concat($"name_1",$"name"))).drop($"name_1").show(false)
+---+---------------------+
|id |name |
+---+---------------------+
|1 |Total: |
|2 |Male: |
|3 |Male:Under 5 years |
|4 |Male:5 to 9 years |
|5 |Male:10 to 14 years |
|6 |Female: |
|7 |Female:Under 5 years |
|8 |Female:5 to 9 years |
|9 |Female:10 to 14 years|
+---+---------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.