使用 Databricks 将从 Azure 服务总线接收到的消息移动到 Azure DataLake
[英]Moving Messages received from Azure Service Bus to Azure DataLake with Databricks
问题描述
I have located a couple of links showing how to Send and Receive messages with Databricks on Apache Spark, included in the following SO question posted sometime ago Structured Streaming with Azure Service Bus Topics
我找到了几个链接，显示如何在 Apache Spark 上使用 Databricks 发送和接收消息，包括在前段时间发布的以下 SO 问题中Structured Streaming with Azure Service Bus Topics
However, I'm struggling to find information on how to create a dataframe from the received messages in order to move the messages to say Azure Data Lake or SQL DB.但是，我正在努力寻找有关如何从收到的消息中创建 dataframe 的信息，以便将消息移动到 Azure Data Lake 或 SQL DB。
Has anyone come across any useful documentation?有没有人遇到任何有用的文档？
Please Note: I'm not referring to Azure Event Hub请注意：我指的不是 Azure 事件中心
1 个解决方案
解决方案1
2 已采纳 2022-03-16 00:01:11
My team and I were looking for the same ability to no avail.我和我的团队一直在寻找同样的能力，但没有成功。 We ended up using an Azure function to execute a python script that leverages pandas and pyarrow to write as parquet to the raw zone of our data lake.我们最终使用 Azure function 来执行 python 脚本，该脚本利用 pandas 和 pyarrow 将 parquet 写入我们数据湖的原始区域。 That ended up causing issues when we tried to read the data with Spark.当我们尝试使用 Spark 读取数据时，这最终导致了问题。
Another approach that looks promising is using an azure function to write the messages as JSON in the raw zone.另一种看起来很有希望的方法是使用 azure function 将消息写入原始区域中的 JSON。 Then use the Auto Loader to process new JSON files.然后使用 Auto Loader 处理新的 JSON 文件。
https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader
There's also this suggestion from Microsoft:微软也有这个建议：
https://learn.microsoft.com/en-us/answers/questions/116689/spark-structured-streaming-azure-service-bus.html https://learn.microsoft.com/en-us/answers/questions/116689/spark-structured-streaming-azure-service-bus.html
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.