并行处理多个数据帧 Scala

Question

I am a newbie in Scala-Spark. I have a dataframe like the one below that I need to split into different chunks of data based into a group ID and process them independently in parallel.

+----+-------+-----+-------+
|user|feature|value|groupID
+----+-------+-----+-------+
|   1|    100|    1|      A|
|   2|    20B|    0|      B|
|   3|    30A|    1|      B|
|   4|    40A|    1|      B| 
|   5|    50A|    1|      A|
|   6|    10A|    0|      B|
|   7|    200|    1|      A|
|   8|    30B|    1|      B|
|   9|    400|    0|      A|
|  10|    50C|    0|      A|
+----+-------+-----+-------+

1 Step I need to split it to have two different df like these ones: I can user a filter for this. But I am not sure if (due to the large number of different dataframes they will produce) I should save them into ADLS as parquets or keep them in memory.

+----+-------+-----+-------+
|user|feature|value|groupID
+----+-------+-----+-------+
|   1|    100|    1|      A|
|   5|    50A|    1|      A|
|   7|    200|    1|      A|
|   9|    400|    0|      A|
|  10|    50C|    0|      A|
+----+-------+-----+-------+

+----+-------+-----+-------+
|user|feature|value|groupID
+----+-------+-----+-------+
|   2|    20B|    0|      B|
|   3|    30A|    1|      B|
|   4|    40A|    1|      B| 
|   6|    10A|    0|      B|
|   8|    30B|    1|      B|
+----+-------+-----+-------+

2 Step Process independently each dataframe in a parallel fashion and get independent processed dataframes.

To give some context:

• The number of groupIds will be high therefore they cannot be hardcoded.

• The processing of each dataframe would ideally happen in parallel.

I ask for a brief idea on how to proceed: I have seen.par.foreach (but is not clear to me how to apply this on a dynamic number of dataframes and how to store them independently nor if the best efficient way)

Answer 1

Check below code.

scala> df.show(false)
+----+-------+-----+-------+
|user|feature|value|groupID|
+----+-------+-----+-------+
|1   |100    |1    |A      |
|2   |20B    |0    |B      |
|3   |30A    |1    |B      |
|4   |40A    |1    |B      |
|5   |50A    |1    |A      |
|6   |10A    |0    |B      |
|7   |200    |1    |A      |
|8   |30B    |1    |B      |
|9   |400    |0    |A      |
|10  |50C    |0    |A      |
+----+-------+-----+-------+

Get distinct groupid values from dataframe.

scala> val groupIds = df.select($"groupID").distinct.as[String].collect // Get distinct group ids.
groupIds: Array[String] = Array(B, A)

Use .par for parallel process. You need add your logic inside map .

scala> groupIds.par.map(groupid => df.filter($"groupId" === lit(groupid))).foreach(_.show(false)) // here you might need add your logic to save or any other inside map function not foreach.., for example I have added logic to show dataframe content in foreach.
+----+-------+-----+-------+
|user|feature|value|groupID|
+----+-------+-----+-------+
|2   |20B    |0    |B      |
|3   |30A    |1    |B      |
|4   |40A    |1    |B      |
|6   |10A    |0    |B      |
|8   |30B    |1    |B      |
+----+-------+-----+-------+

+----+-------+-----+-------+
|user|feature|value|groupID|
+----+-------+-----+-------+
|1   |100    |1    |A      |
|5   |50A    |1    |A      |
|7   |200    |1    |A      |
|9   |400    |0    |A      |
|10  |50C    |0    |A      |
+----+-------+-----+-------+