在使用 pyspark 读取镶木地板的情况下提供模式有什么意义？

Question

When I am working with CSV, I can provide custom schema while reading a file, and the benefits I receive are as follows (along with the contrast with the parquet case):

1. All the rows do not have to be scanned through to infer the schema. (Parquet: Since the schema is stored in a separate file inferring schema is as simple as reading the schema from this file)
2. I can convert the schema of the file on the fly. For example, if I have stored integer numbers as a string(numbers) then I can provide the schema to read that column as an integer and it does. (This is impossible in case of parquet since it throws the Expected: IntegerType, found: binary exception in case the schema provided is of integer and the parquet column is stored as a string).

Given these two points, what are the benefits of providing schema in case of the parquet files?

Answer 1

Apart from the obvious benefit of type safety (that you've already mentioned;)), providing a schema on read allows parquet data source to perform schema merging , for one:

scala> val df2 = spark.range(1,6).map(i => (2,i,i*i)).toDF("version","value","square")
df2: org.apache.spark.sql.DataFrame = [version: int, value: bigint ... 1 more field]

scala> df2.write.parquet("/tmp/reconciled")

scala> val df3 = spark.range(1,6).map(i => (3,i,i*i*i)).toDF("version","value","cube")
df3: org.apache.spark.sql.DataFrame = [version: int, value: bigint ... 1 more field]

scala> df3.write.mode("append").parquet("/tmp/reconciled")

scala> val sch = StructType(Array(StructField("version",IntegerType,false),
     | StructField("value",LongType,true),
     | StructField("square",LongType,false),
     | StructField("cube",LongType,false)))
sch: org.apache.spark.sql.types.StructType = StructType(StructField(version,IntegerType,false), StructField(value,LongType,true), StructField(square,LongType,false), StructField(cube,LongType,false))

scala> val dfn = spark.read.schema(sch).parquet("/tmp/reconciled")
dfn: org.apache.spark.sql.DataFrame = [version: int, value: bigint ... 2 more fields]

scala> dfn.show(false)
+-------+-----+------+----+                                                     
|version|value|square|cube|
+-------+-----+------+----+
|2      |1    |1     |null|
|2      |2    |4     |null|
|2      |3    |9     |null|
|2      |4    |16    |null|
|2      |5    |25    |null|
|3      |1    |null  |1   |
|3      |2    |null  |8   |
|3      |3    |null  |27  |
|3      |4    |null  |64  |
|3      |5    |null  |125 |
+-------+-----+------+----+

...so that you can process a range of similar but different parquet files as if they all had the same schema.