for循环中Spark列的数据类型验证-Spark DataFrame

Question

I'm trying to validate datatype of DataFrame before entering the loop, wherein I'm trying to do SQL calculation, but datatype validation is not going through and it is not getting inside the loop. The operation needs to be performed on only numeric columns.

How can this be solved? Is this the right way to handle datatype validation?

//get datatype of dataframe fields
val datatypes =  parquetRDD_subset.schema.fields

//check if datatype of column is String and enter the loop for calculations.

for (val_datatype <- datatypes if val_datatype.dataType =="StringType") 
{
    val dfs = x.map(field => spark.sql(s"select * from table"))
    val withSum = dfs.reduce((x, y) => x.union(y)).distinct()
}

Answer 1

You are comparing the dataType to a string which will never be true (for me the compiler complains that they are unrelated). dataType is an object which is a subtype of org.apache.spark.sql.types.DataType.

Try replacing your for with

for (val_datatype <- datatypes if val_datatype.dataType.isInstanceOf[StringType]) 

In any case, your for loop does nothing but declare the vals, it doesn't do anything with them.