How to send Kafka message from one topic to another topic?

Question

suppose my producer is writing the message to Topic A...once the message is in Topic A, i want to copy the same message to Topic B. Is this possible in kafka?

Answer 1

If I understand correctly, you just want stream.to("topic-b") , although, that seems strange without doing something to the data.

Note:

The specified topic should be manually created before it is used

Answer 2

I am not clear about what use case you are exactly trying to achieve by simply copying data from one topic to another topic. If both the topics are in the same Kafka cluster then it is never a good idea to have two topics with the same message/content.

I believe the gap here is that probably you are not clear about the concept of the Consumer group in Kafka . Probably you have two action items to do by consuming the message from the Kafka topic. And you are believing that if the first application consumes the message from the Kafka topic, will it be available for the second application to consume the same message or not. Kafka allows you to solve this kind of common use case with the help of the consumer group.

Let's try to differentiate between other message queue and Kafka and you will understand that you do not need to copy the same data/message between two topics.

In other message queues, like SQS(Simple Queue Service) where if the message is consumed by a consumer, the same message is not available to get consumed by other consumers. It is the responsibility of the consumer to delete the message safely once it has processed the message. By doing this we guarantee that the same message should not get processed by two consumers leading to inconsistency.

But, In Kafka, it is totally fine to have multiple sets of consumers consuming from the same topic. The set of consumers form a group commonly termed as the consumer group. Here one of the consumers from the consumer group can process the message based on the partition of the Kafka topic the message is getting consumed from.

Now the catch here is that we can have multiple consumer groups consuming from the same Kafka topic . Each consumer group will process the message in the way they want to do. There is no interference between consumers of two different consumer groups .

To fulfill your use case I believe you might need two consumer groups that can simply process the message in the way they want. You do not essentially have to copy the data between two topics.

Hope this helps.

Answer 3

There are two immediate options to forward the contents of one topic to another:

1. by using the stream feature of Kafka to create a forwarding link between the two topics.
2. by creating a consumer / producer pair and using those to receive and then forward on messages

I have a short piece of code that shows both (in Scala):

  def topologyPlan(): StreamsBuilder = {
    val builder = new StreamsBuilder
    val inputTopic: KStream[String, String] = builder.stream[String, String]("topic2")
    inputTopic.to("topic3")
    builder
  }

  def run() = {
    val kafkaStreams = createStreams(topologyPlan())
    kafkaStreams.start()

    val kafkaConsumer = createConsumer()
    val kafkaProducer = createProducer()
    kafkaConsumer.subscribe(List("topic1").asJava)
    while (true) {
      val record = kafkaConsumer.poll(Duration.ofSeconds(5)).asScala
      for (data <- record.iterator) {
        kafkaProducer.send(new ProducerRecord[String, String]("topic2", data.value()))
      }
    }
  }

Looking at the run method, the first two lines set up a streams object to that uses the topologyPlan() to listen for messages in 'topic2' and forward then to 'topic3'.

The remaining lines show how a consumer can listen to a 'topic1' and use a producer to send them onward to 'topic2'.

The final point of the example here is Kafka is flexible enough to let you mix options depending on what you need, so the code above will take messages in 'topic1', and send them to 'topic3' via 'topic2'.

If you want to see the code that sets up consumer, producer and streams, see the full class here .