Spark Scala | 动态创建数据框

Question

I would like to create dataframe names dynamically from a collection.

Please see below:

val set1 = Set("category1","category2","category3")

The following is a UDF which takes a string x from the set as input and generate the dataframe accordingly:

def catDfgen(x: String): DataFrame = {
    spark.sql(s"select * from table where col1 = '$x'")
}

Now I need help here, to create not only DataFrame but also the DataFrame name should be dynamically generated in order to achieve

val category1DF = catDfgen($x)
val category2DF = catDfgen($x)

...etc. Would it be possible to do it using the code below?

set1.map( x =>  val $x+"DF" = catDfgen($x))

If not please suggest an effective method.

Answer 1

Suman, I believe the below might help your use-case

import org.apache.spark.sql.{DataFrame, SparkSession}

object Test extends App {

  val spark: SparkSession = SparkSession.builder().master("local").getOrCreate()

  val set1 = Set("category1","category2","category3")

  val dfs: Map[String, DataFrame] = set1.map(x =>
    (s"${x}DF", spark.sql(s"select * from table where col1 = '$x'").alias(s"${x}DF").toDF())
  ).toMap

  dfs("category1DF").show()

  spark.stop()
}