How to populate a new spark dataframe column with values from specific rows in another column. Need suggestions

Question

My issue is this:

I have a spark dataframe that looks like this
+-----------+---------------+
|         id|           name|
+-----------+---------------+
|          1|         Total:|
|          2|          Male:|
|          3|  Under 5 years|
|          4|   5 to 9 years|
|          5| 10 to 14 years|
|          6|        Female:|
|          7|  Under 5 years|
|          8|   5 to 9 years|
|          9| 10 to 14 years|
+-----------+---------------+

I want to create a new DF with an added column that will look like this:
+-----------+---------------+---------------------+
|         id|           name|             new_name|
+-----------+---------------+---------------------+
|          1|         Total:|               Total:|
|          2|          Male:|                Male:|
|          3|  Under 5 years|  Male: Under 5 years|
|          4|   5 to 9 years|  Male: Under 5 years|
|          5| 10 to 14 years|  Male: Under 5 years|
|          6|        Female:|              Female:|
|          7|  Under 5 years|Female: Under 5 years|
|          8|   5 to 9 years|Female: Under 5 years|
|          9| 10 to 14 years|Female: Under 5 years|
+-----------+---------------+---------------------+

I don't have any code worth showing I'm looking for ways to approach the problem. I assume it would be something like:

val dfB = dfA.withColum(row => aUDF(row))

I'm assuming the solution will need some kind of UDF. I assume it needs to loop or map and update a "prefix" val any time it finds a row with ":" in the name field. But I don't know how to go about doing that. Any ideas would be much appreciated.

Answer 1

Spark 2.4.3 you can achieve this by using split and last window function.

scala> import org.apache.spark.sql.expressions.Window
scala> var df = spark.createDataFrame(Seq((1,"Total:"), (2,"Male:"),(3,  "Under 5 years"),(4,"5 to 9 years"),(5, "10 to 14 years"),(6,"Female:"),(7,"Under 5 years"),(8,"5 to 9 years"),(9, "10 to 14 years"))).toDF("id","name")

scala> df.show

+---+--------------+
| id|          name|
+---+--------------+
|  1|        Total:|
|  2|         Male:|
|  3| Under 5 years|
|  4|  5 to 9 years|
|  5|10 to 14 years|
|  6|       Female:|
|  7| Under 5 years|
|  8|  5 to 9 years|
|  9|10 to 14 years|
+---+--------------+
scala>  var win =Window.orderBy(col("id"))    

scala> var df2 =df.withColumn("name_1",last(when(split($"name",":")(1) ==="",$"name"),true).over(win))

scala> df2.withColumn("name",when($"name"===$"name_1",$"name").otherwise(concat($"name_1",$"name"))).drop($"name_1").show(false)
+---+---------------------+
|id |name                 |
+---+---------------------+
|1  |Total:               |
|2  |Male:                |
|3  |Male:Under 5 years   |
|4  |Male:5 to 9 years    |
|5  |Male:10 to 14 years  |
|6  |Female:              |
|7  |Female:Under 5 years |
|8  |Female:5 to 9 years  |
|9  |Female:10 to 14 years|
+---+---------------------+