using caseclass versus structtype in spark scala

Question

When should I use Structtype, and when should I use case class. I am trying to create a spark dataset. I have an input CSV file, I am trying to create a dataframe first and then convert it to the dataset using df.as[]. Now in order to generate the schema, should I use structtype or case class? Please help.

Answer 1

You don't have to use StructType when reading your CSV file but :

• By default all fields would be Strings unless you specify the inferschema option

• You'd have to name every field like this if you don't have a header

  sparkSession.read.csv("my/csv/path.csv").toDF("id","product","customer","time").as[Transaction]