将Spark架构从Dataframe与T类型进行比较

Question

I am trying to add some runtime type checks when writting a Spark Dataframe, basically I want to make sure that the DataFrame schema is compatible with a type T, compatible doesn't mean that it has to be exactly the same. Here is my code

def save[T: Encoder](dataframe: DataFrame, url: String): Unit = {
        val encoder = implicitly[Encoder[T]]
        assert(dataframe.schema == encoder.schema, s"Unable to save schemas don't match")

        dataframe.write.parquet(url)
      }

Currently I am checking that the schemas are equals, how could I check that they are compatible with the type T?

With compatible I mean that if I execute dataframe.as[T] it will work (but I don't want to execute that because it is quite expensive)

Answer 1

Create an empty dataframe with the same schema and call .as[T] on it. If it works the schema should be compatible!