使用 spark 写入镶木地板文件时如何添加额外的元数据

Question

Looks like spark by default write "org.apache.spark.sql.parquet.row.metadata" to parquet file footer. However, what if I want to write some random metadata(such as version=123) to a parquet file produced by spark?

This does NOT work:

df.write().option("version","123").parquet("somefile.parquet");

And I'm using spark version 1.6.2

Answer 1

Column level metadata, yes see my comment.

Table level comments/user metadata: See https://issues.apache.org/jira/browse/SPARK-10803

Sadly, not yet