[英]Spark Structured Streaming: path not specified error
Trying to write data to Kafka topic using Spark Structured Streaming and getting following error. 尝试使用Spark结构化流将数据写入Kafka主题,并得到以下错误。
aggregatedDataset
.select(to_json(struct("*")).as("value"))
.writeStream()
.outputMode(OutputMode.Append())
.option("kafka.bootstrap.servers", kafkaBootstrapServersString)
.option("topic", topic)
.option("checkpointLocation", checkpointLocation)
.start();
Stacktrace: 堆栈跟踪:
Exception in thread "main" java.lang.IllegalArgumentException: 'path' is not specified
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$11.apply(DataSource.scala:276)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$11.apply(DataSource.scala:276)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at org.apache.spark.sql.catalyst.util.CaseInsensitiveMap.getOrElse(CaseInsensitiveMap.scala:28)
at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:275)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
In your writeStream section format is missing, which in you case seems to be kafka, 在您的writeStream部分中缺少格式,在您看来,该格式似乎是kafka,
aggregatedDataset
...
.writeStream
.format("kafka")
...
Hope this helps! 希望这可以帮助!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.