[英]Databricks: Azure Queue Storage structured streaming key not found error
I am trying to write ETL pipeline for AQS streaming data.我正在尝试为 AQS 流数据编写 ETL 管道。 Here is my code这是我的代码
CONN_STR = dbutils.secrets.get(scope="kvscope", key = "AZURE-STORAGE-CONN-STR")
schema = StructType([
StructField("id", IntegerType()),
StructField("parkingId", IntegerType()),
StructField("capacity", IntegerType()),
StructField("freePlaces", IntegerType()),
StructField("insertTime", TimestampType())
])
stream = spark.readStream \
.format("abs-aqs") \
.option("fileFormat", "json") \
.option("queueName", "freeparkingplaces") \
.option("connectionString", CONN_STR) \
.schema(schema) \
.load()
display(stream)
When I run this I am getting java.util.NoSuchElementException: key not found: eventType
当我运行它时,我得到java.util.NoSuchElementException: key not found: eventType
Here is how my queue looks like这是我的队列的样子
Can you spot and explain me what is the problem?你能发现并解释一下问题是什么吗?
The abs-aqs
connector isn't for consumption of data from AQS, but it's for getting data about new files in the blob storage using events reported to AQS. abs-aqs
连接器不是用于使用 AQS 中的数据,而是用于使用报告给 AQS 的事件获取有关 blob 存储中新文件的数据。 That's why you're specifying the the file format option, and schema - but these parameters will be applied to the files, not messages in AQS.这就是您指定文件格式选项和架构的原因 - 但这些参数将应用于文件,而不是 AQS 中的消息。
As far as I know (I could be wrong), there is no Spark connector for AQS, and it's usually recommended to use EventHubs or Kafka as messaging solution.据我所知(我可能是错的),AQS 没有 Spark 连接器,通常建议使用 EventHubs 或 Kafka 作为消息传递解决方案。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.