Is it possible to consume from an Azure EventHub topic using Apache Beam / Google Cloud Dataflow?

Question

Problem

We'd like to consume from an EventHub topic in Azure using a dataflow pipeline in Google Cloud.

Question

Is it known whether KafkaIO allows for consuming from EventHubs directly in an Apache Beam/Google Cloud dataflow job? (see this post )

Alternative approaches for getting the EventHub data into Pub/Sub are also appreciated (eg Publish from Azure Streaming Analytics to Pub/Sub)

Thank you!

Answer 1

Azure Event Hubs supports the Apache Kafka protocol 1.0 and later , so you should be able to consume events from Event Hubs using KafkaIO (which supports Kafka versions 0.10.1 and newer). Google Cloud has a guide for processing messages from Kafka in Dataflow here .

Answer 2

Azure 事件中心与 Kafka 协议二进制兼容，因此任何 Kafka 客户端或管道都应该能够使用来自事件中心主题的事件。