需要从 spark-shell 将 json dataframe 写入 avro 文件格式

Question

嗨，我必须读取 json 数据来触发 DF，然后我需要从 spark shell 以 avro 文件格式写入 DF，我收到以下错误：

org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".;
  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:647)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:245)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
  ... 49 elided

Answer 1

您是否尝试过添加 spark-avro 库？

您可以在启动 spark-shell 时执行以下操作：

spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.6

Answer 2

spark-avro外部库，您必须将spark-avro package 导入spark-shell 。

检查下面

spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.0