“Pushing” Messages From One Kafka Topic To Another

Question

So, I have been tasked with setting up a kafka to kafka streaming service, whereby my company will send kafka messages from our topics, to a customers kafka topic.

What would be the best tool for this job?

Answer 1

If you are looking for replication then you should be looking at mirror maker. https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0

If you don't control the source of destination end point then you might need to do more with access control. The question is a little under specified, but in any case looking at what mirror maker does and how it works is a good starting point.

If you need something more comprehensive that will work with sources or destinations that are not Kafka then this is something you might want to look at. https://github.com/linkedin/brooklin