DataFrame Scala上的DataType验证

Question

I need to validate datatypes of DataFrame.

Sample DF

val rawData = Seq((1,"First Rec Col 1" , "First Rec Col 2" ), (1,"Second Rec Col 1" , "Second Rec Col 2")).toDF("Raw_PK" ,"Col1", "Col2")

rawData.show

Result :

Here is my schema,

val types = Seq(("Col1", "string"), ("Col2", "double"))

It says Col1 should be a String type and Col2 should be double

What I have tried ?

There are couple of ways i tried (traditional way of looping) but want to get rid of that. So here is what i did

val df2 = rawData.select(types.map{case (c, t) => col(c).cast(t)} : _*)
df2.show

It is trying to cast Col2 String to Double , It showed null in Col2

I want to achieve that it should add ANOTHER COLUMN saying its not a valid record to process.

Any help ? Thanks in advance.

Answer 1

You can use the techniques described here: https://gist.github.com/dennyglee/c21f59cf81216c1dc9a38525a0e41de1

DataType verification on DataFrame Scala

Using pattern matching:

import org.apache.spark.sql.types.IntegerType

assert(testDF.schema(col1).dataType match {
  case IntegerType => true
  case _ => false
})