根据Apache Spark Scala中的列数据类型将数据框中的列选择为另一个数据框

Question

I have a spark dataframe

inputDF: org.apache.spark.sql.DataFrame = [_id: string, Frequency:              double, Monterary: double, Recency: double, CustID: string]
        root
     |-- _id: string (nullable = false)
     |-- Frequency: double (nullable = false)
     |-- Monterary: double (nullable = false)
     |-- Recency: double (nullable = false)
     |-- CustID: string (nullable = false)

I want to create a new dataframe by dropping string columns from this. Specific condition is not to iterate over the column names . Anyone has any idea ?

Answer 1

If schema is flat and contains only simple types you can filter over fields but unless you have a crystal ball you cannot really avoid iteration:

import org.apache.spark.sql.types.StringType
import org.apache.spark.sql.functions.col

df.select(df.schema.fields.flatMap(f => f.dataType match {
  case StringType => Nil
  case _ => col(f.name) :: Nil
}): _*)