[英]How to populate a new spark dataframe column with values from specific rows in another column. Need suggestions
My issue is this:我的问题是这样的:
I have a spark dataframe that looks like this
+-----------+---------------+
| id| name|
+-----------+---------------+
| 1| Total:|
| 2| Male:|
| 3| Under 5 years|
| 4| 5 to 9 years|
| 5| 10 to 14 years|
| 6| Female:|
| 7| Under 5 years|
| 8| 5 to 9 years|
| 9| 10 to 14 years|
+-----------+---------------+
I want to create a new DF with an added column that will look like this:
+-----------+---------------+---------------------+
| id| name| new_name|
+-----------+---------------+---------------------+
| 1| Total:| Total:|
| 2| Male:| Male:|
| 3| Under 5 years| Male: Under 5 years|
| 4| 5 to 9 years| Male: Under 5 years|
| 5| 10 to 14 years| Male: Under 5 years|
| 6| Female:| Female:|
| 7| Under 5 years|Female: Under 5 years|
| 8| 5 to 9 years|Female: Under 5 years|
| 9| 10 to 14 years|Female: Under 5 years|
+-----------+---------------+---------------------+
I don't have any code worth showing I'm looking for ways to approach the problem.我没有任何值得展示的代码我正在寻找解决问题的方法。 I assume it would be something like:我认为它会是这样的:
val dfB = dfA.withColum(row => aUDF(row))
I'm assuming the solution will need some kind of UDF.我假设解决方案需要某种 UDF。 I assume it needs to loop or map and update a "prefix" val any time it finds a row with ":" in the name field.我假设它需要循环或映射并在任何时候在名称字段中找到带有“:”的行时更新“前缀”val。 But I don't know how to go about doing that.但我不知道该怎么做。 Any ideas would be much appreciated.任何想法将不胜感激。
Spark 2.4.3 you can achieve this by using split and last window function. Spark 2.4.3 你可以通过使用 split 和 last window 函数来实现这一点。
scala> import org.apache.spark.sql.expressions.Window
scala> var df = spark.createDataFrame(Seq((1,"Total:"), (2,"Male:"),(3, "Under 5 years"),(4,"5 to 9 years"),(5, "10 to 14 years"),(6,"Female:"),(7,"Under 5 years"),(8,"5 to 9 years"),(9, "10 to 14 years"))).toDF("id","name")
scala> df.show
+---+--------------+
| id| name|
+---+--------------+
| 1| Total:|
| 2| Male:|
| 3| Under 5 years|
| 4| 5 to 9 years|
| 5|10 to 14 years|
| 6| Female:|
| 7| Under 5 years|
| 8| 5 to 9 years|
| 9|10 to 14 years|
+---+--------------+
scala> var win =Window.orderBy(col("id"))
scala> var df2 =df.withColumn("name_1",last(when(split($"name",":")(1) ==="",$"name"),true).over(win))
scala> df2.withColumn("name",when($"name"===$"name_1",$"name").otherwise(concat($"name_1",$"name"))).drop($"name_1").show(false)
+---+---------------------+
|id |name |
+---+---------------------+
|1 |Total: |
|2 |Male: |
|3 |Male:Under 5 years |
|4 |Male:5 to 9 years |
|5 |Male:10 to 14 years |
|6 |Female: |
|7 |Female:Under 5 years |
|8 |Female:5 to 9 years |
|9 |Female:10 to 14 years|
+---+---------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.