如何加入两个具有多个分区的Kafka流？

Question

I have two Kafka streams, request and event each partitioned on a common field requestId (last two digits). I want to join both the streams and write to HDFS or Local filesystem ? How to write an efficient consumer which considers only the relevant partitions while joining the two streams ?

Answer 1

You should use Kafka's Streams API , Apache Kafka's stream processing library, instead of a hand written consumer. To write the data to HDFS you should use Kafka Connect.

For doing the join, look at this question: How to manage Kafka KStream to Kstream windowed join?

Also check out Confluent's documentation about Kafka Streams and Kafka Connect to get started. If you have further question, please start a follow up question (after reading the manual :))

Answer 2

Kafka streams with Kafka Connect (for HDFS) is a straightforward solution. However, it must be pointed out that the HDFS connector for Kafka Connect is only available with the Confluent's version of Kafka. The Apache Kafka Connect only comes with a file writer and not HDFS writer.