Moving Messages received from Azure Service Bus to Azure DataLake with Databricks

Question

I have located a couple of links showing how to Send and Receive messages with Databricks on Apache Spark, included in the following SO question posted sometime ago Structured Streaming with Azure Service Bus Topics

However, I'm struggling to find information on how to create a dataframe from the received messages in order to move the messages to say Azure Data Lake or SQL DB.

Has anyone come across any useful documentation?

Please Note: I'm not referring to Azure Event Hub

Answer 1

My team and I were looking for the same ability to no avail. We ended up using an Azure function to execute a python script that leverages pandas and pyarrow to write as parquet to the raw zone of our data lake. That ended up causing issues when we tried to read the data with Spark.

Another approach that looks promising is using an azure function to write the messages as JSON in the raw zone. Then use the Auto Loader to process new JSON files.

https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader

There's also this suggestion from Microsoft:

https://learn.microsoft.com/en-us/answers/questions/116689/spark-structured-streaming-azure-service-bus.html