如何从 Azure Databricks 将 JSON 写入 Azure 队列

Question

I'm trying to read a JSON file from BLOB and write that file in Azure queue. The reading part works fine but while writing it throws an error.

I've already tried the URL of the queue folder in which I'm trying to write, as parameter for.save()

Here's my code:

storage_account_name="mrktmabcdestaaue"
storage_account_access_key="myurl=="
file_location="wasbs://myfolder@mrktmabcdestaaue.blob.core.windows.net/input.json"
file_type="json"
spark.conf.set(
        "fs.azure.account.key."+storage_account_name+".blob.core.windows.net",
  storage_account_access_key)
df = spark.read.option("multiline", "true").format(file_type).load(file_location)

df.write.mode("overwrite").format("com.databricks.spark.json").save("wasbs://myqueue@mrktmabcdestaaue.queue.core.windows.net")

My Input Json:

{
"Name": "Abc",
"Age": 18,
"City": "def"
}

The error message I'm getting is:

"shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container myqueue in account mrktmabcdestaaue.queue.core.windows.net using anonymous credentials, and no credentials found for them in the configuration."

Answer 1

It sounds like your cluster is not attached to that storage account. Recreate your cluster and make sure that the account is attached to the cluster.

Your storage location should be wasbs://myfolder@mrktmabcdestaaue.blob.core.windows.net/input.json .

For more details, you could refer to this article .

Answer 2

This scenario is not supported. You can write to Blob Storage Container, but not to Storage Queues.

The Databricks Azure Queue (AQS) connector uses Azure Queue Storage (AQS) to provide an optimized file source that lets you find new files written to an Azure Blob Storage (ABS) container without repeatedly listing all of the files. See the Documentation for more details. So can also only be used reading files.

If you want to pass the content you read to consumers you could use Azure EventHubs or Apache Kafka (on Azure HDInsight or Confluent) as a message broker. In this scenario you would make use of structured streaming. So you have to have a streaming data frame. Writing back stream would look like this:

df \
.writeStream \
.format("eventhubs") \
.options(**ehConf) \
.option("checkpointLocation", checkploc) \
.start()

Ohter option could be to make use of Azure Event Grid. Finally it depends on the concrete scenario you would like to achieve.

Answer 3

I had the same scenario, tried the same thing, ended up here as a result.

I also tried dbutils.fs.mount but was not able to mount the storage queue as I could with blob storage.

I ended up using a storage queue client:
https://pypi.org/project/azure-storage-queue/

Then I had to read each json message and call queue.send_message for each one. Not great but could not find a better solution.