尝试将 Azure Databricks 与 Azure 事件中心连接，以将一些属性发送到事件中心并通过 splunk 读取

Question

I am looking to connect Azure data bricks to Event hub and read it through splunk. Initially I was able to send a test message and was able to receive the events in splunk(It was possible using scala as per https://docs.microsoft.com/en-us/azure/databricks/scenarios/databricks-stream-from-eventhubs --> Send tweets to event hubs). Now I am trying to implement the same using python using the reference from https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send ---> Send Event. When I try to pass the object-parameters, it throws an error like unable to import from Azure Event hubs

Can anyone help me understand how can I connect Azure databricks with Azure Eventhubs and include sending object -parameters?

PS: I have added the necessary libraries required to the cluster as below:

1. com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.18
2. azure-eventhubs-spark_2.11
3. com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22
4. org.apache.spark:spark-avro_2.12:3.1.1

I have checked it with different versions of libraries too.

Can someone help me with the syntax part on how to pass the object parameters in the format of key value pairs?

Answer 1

It's simple to connect to EventHubs from Azure Databricks - just follow official documentation, specifically the section Writing Data to Event Hubs (example is for batch write):

writeConnectionString = "SharedAccessSignatureFromAzurePortal"
ehWriteConf = {
  'eventhubs.connectionString' : 
     sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(writeConnectionString)
}

# Write body data from a DataFrame to EventHubs. 
# Events are distributed across partitions using round-robin model.
ds = df \
  .select("body") \
  .write \
  .format("eventhubs") \
  .options(**ehWriteConf) \
  .save()

You need to construct the body column somehow - by encoding your data as JSON using the to_json(struct("*")) , or encoding data as Avro...

But you also have a problem in your cluster configuration - specifically these two libraries: azure-eventhubs-spark_2.11 and com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22 - they are for Spark 2, but you use Spark 3. Uninstall them, and restart the cluster.