Spark DataFrame 和 Dataset 之间的编码器差异

Question

When reading about the differences between Spark's DataFrame (which is an alias for Dataset[Row] ) and Dataset , it's often mentioned that Dataset s make use of Encoders to efficiently convert to/from JVM objects to Spark's internal data representation. In scala, there are implicit encoders provided for case classes and primitive types. However, there is also the RowEncoder which, I believe, achieves encoding for the Row in DataFrames.

My questions are

1. In terms of efficient conversion between JVM objects and Spark's internal binary representation, are DataFrame s and Dataset s the same in performance?
2. What additional benefits do a specific type (like a case class in Scala) provide over the generic Row as far as Encoding (serializing/deserializing) goes? Apart from compile-time type-safety, do typed JVM objects provide any advantage over semi-typed (or "untyped") Row ?

Answer 1

Dataframes are just datasets with an encoder for the Spark Row class. So in essence a Dataframe is a Dataset.

Encoders also do not come into play at all unless you are using non column functions (functions that take a lambda like map, reduce, flatmap.) The moment that you do use one of those functions there will be a performance hit as you break the codegen that catalyst is doing into two parts since it can't optimize the lambda. This means your probably don't want to be using those functions at all and can ignore the dataset/Dataframe difference entirely since if you don't use those functions you won't ever encode.

In my experience the benefit of the type safety you can get with a Dataset and the types apis is not worth the huge perf hit. In almost all cases I've found that you should stay in Dataframes and only use column based functions and udfs for best performance.

As an additional note, the only other time an encoder will be used is when you parallelize a collection, all Datasources will provide Rows or Internal rows to Spark so your encoder will not be used for most sources.