从一个集群到多个集群的KStream

Question

I need to have one java module in my Kafka cluster that will get a message, and depending on two fields of the message, it would be enriched and published to another kafka cluster. I've been researching KStream API and I can't find a way to send things to another cluster (without using the publisher library).

Any ideas and how can I do this?

Thanks!

Answer 1

I need to have one java module in my Kafka cluster that will get a message, and depending on two fields of the message, it would be enriched and published to another kafka cluster. I've been researching KStream API and I can't find a way to send things to another cluster (without using the publisher library).

At the moment, the Kafka's Streams API interacts only with one given cluster at a time. That is, you cannot read from Kafka cluster A and then write to Kafka cluster B with the Streams API.

Any ideas and how can I do this?

I'd therefore recommend to use Kafka Connect for this. The latest versions of Kafka include the so-called Single Message Transformations feature in Kafka Connect, which you can use for such routing of messages.

If you are open to use a commercial tool, Confluent Replicator ( docs ) allows you to do such cross-cluster data flows, as Hans Jespersen mentioned above.

A good high-level starting point would be the talk Single Message Transformations Are Not the Transformations You're Looking For (don't be deterred by the title :-P) by Ewen Cheslack-Postava, one of the Kafka committers who developed the new SMT feature. There's also a blog post that briefly describes SMT in Kafka Connect.