如何在 Spark Scala 的 For 循环中动态创建多个数据帧

Question

I have a Array[string] which has the value ["DF1" , "DF2", "DF3"]

I have values like below in my config file.

DF1=select * from TB1
DF2=select * from TB2
DF3=select * from TB3

I need to create 3 Data frames dynamically in a For loop by unloading the respective tables.

lets say my Array name is ARRAY1.

My code would be

for (for1 <- ARRAY1){val $for1+_DF = spark.sql(for1)}

Above is just kind of pseudo code.

please help me by providing the correct code syntax.

Thanks, Naveen

Answer 1

Check below code.

scala> val queries = Map(
                          "df1" -> "select * from tb1",
                          "df2" -> "select * from TB2",
                          "df3" -> "select * from TB3"
                      ) // After reading from config file.

scala> 

queries
.values
.par
.map(spark.sql)
.foreach(_.show(false)) 

// Used .par for parallel loading & all three DataFrame object will be in list & you can do any operation on that, here i am using show function to display values

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+

+---+
|id |
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
|6  |
|7  |
|8  |
|9  |
+---+