如何动态获取Spark数据框中的列的数据类型

Question

I have a dataframe - converted dtypes to map.

val dfTypesMap:Map[String,String]] = df.dtypes.toMap

Output:

(PRODUCT_ID,StringType)
(PRODUCT_ID_BSTP_MAP,MapType(StringType,IntegerType,false))
(PRODUCT_ID_CAT_MAP,MapType(StringType,StringType,true))
(PRODUCT_ID_FETR_MAP_END_FR,ArrayType(StringType,true))

When I use type [String] hardcoding in row.getAS[String] , there is no compilation error.

df.foreach(row => {
  val prdValue = row.getAs[String]("PRODUCT_ID")
})

I want to iterate above map dfTypesMap and get corresponding value type. Is there any way to convert dt column types to general types like below?

StringType --> String
MapType(StringType,IntegerType,false) ---> Map[String,Int]
MapType(StringType,StringType,true) ---> Map[String,String]
ArrayType(StringType,true) ---> List[String]

Answer 1

As mentioned, Datasets make it easier to work with types. Dataset is basically a collection of strongly-typed JVM objects.

You can map your data to case classes like so

case class Foo(PRODUCT_ID: String, PRODUCT_NAME: String)
val ds: Dataset[Foo] = df.as[Foo]

Then you can safely operate on your typed objects. In your case you could do

ds.foreach(foo => {
  val prdValue = foo.PRODUCT_ID
})

For more on Datasets, check out https://spark.apache.org/docs/latest/sql-programming-guide.html#creating-datasets