[英]How to send Kafka message from one topic to another topic?
suppose my producer is writing the message to Topic A...once the message is in Topic A, i want to copy the same message to Topic B. Is this possible in kafka?假设我的制作人正在将消息写入主题 A...一旦消息在主题 A 中,我想将相同的消息复制到主题 B。这在 kafka 中可能吗?
If I understand correctly, you just want stream.to("topic-b")
, although, that seems strange without doing something to the data.如果我理解正确的话,您只需要stream.to("topic-b")
,不过,如果不对数据做任何事情,这似乎很奇怪。
Note:笔记:
The specified topic should be manually created before it is used指定主题在使用前需要手动创建
I am not clear about what use case you are exactly trying to achieve by simply copying data from one topic to another topic.我不清楚您究竟想通过简单地将数据从一个主题复制到另一个主题来实现什么用例。 If both the topics are in the same Kafka cluster then it is never a good idea to have two topics with the same message/content.如果两个主题都在同一个 Kafka 集群中,那么让两个主题具有相同的消息/内容绝不是一个好主意。
I believe the gap here is that probably you are not clear about the concept of the Consumer group in Kafka .我相信这里的差距可能是你对 Kafka 中 Consumer group的概念不太清楚。 Probably you have two action items to do by consuming the message from the Kafka topic.通过使用来自 Kafka 主题的消息,您可能有两个操作项要做。 And you are believing that if the first application consumes the message from the Kafka topic, will it be available for the second application to consume the same message or not.并且您相信如果第一个应用程序使用来自 Kafka 主题的消息,第二个应用程序是否可以使用相同的消息。 Kafka allows you to solve this kind of common use case with the help of the consumer group. Kafka 允许您在消费者组的帮助下解决这种常见用例。
Let's try to differentiate between other message queue and Kafka and you will understand that you do not need to copy the same data/message between two topics.让我们尝试区分其他消息队列和 Kafka,您会明白您不需要在两个主题之间复制相同的数据/消息。
In other message queues, like SQS(Simple Queue Service) where if the message is consumed by a consumer, the same message is not available to get consumed by other consumers.在其他消息队列中,如 SQS(简单队列服务),如果消息被消费者消费,则同一消息无法被其他消费者消费。 It is the responsibility of the consumer to delete the message safely once it has processed the message.消费者有责任在处理完消息后安全地删除消息。 By doing this we guarantee that the same message should not get processed by two consumers leading to inconsistency.通过这样做,我们保证相同的消息不应该被两个消费者处理导致不一致。
But, In Kafka, it is totally fine to have multiple sets of consumers consuming from the same topic.但是,在 Kafka 中,让多组消费者从同一个主题消费是完全没问题的。 The set of consumers form a group commonly termed as the consumer group.一组消费者形成一个通常称为消费者组的组。 Here one of the consumers from the consumer group can process the message based on the partition of the Kafka topic the message is getting consumed from.这里来自消费者组的消费者之一可以根据消息从中消费的 Kafka 主题的分区来处理消息。
Now the catch here is that we can have multiple consumer groups consuming from the same Kafka topic .现在要注意的是,我们可以让多个消费者组消费同一个 Kafka 主题。 Each consumer group will process the message in the way they want to do.每个消费者组将按照他们想要的方式处理消息。 There is no interference between consumers of two different consumer groups .两个不同消费群体的消费者之间没有干扰。
To fulfill your use case I believe you might need two consumer groups that can simply process the message in the way they want.为了实现您的用例,我相信您可能需要两个可以按照他们想要的方式简单地处理消息的消费者群体。 You do not essentially have to copy the data between two topics.您基本上不必在两个主题之间复制数据。
Hope this helps.希望这可以帮助。
There are two immediate options to forward the contents of one topic to another:将一个主题的内容转发到另一个主题有两个直接选项:
I have a short piece of code that shows both (in Scala):我有一小段代码显示了两者(在 Scala 中):
def topologyPlan(): StreamsBuilder = {
val builder = new StreamsBuilder
val inputTopic: KStream[String, String] = builder.stream[String, String]("topic2")
inputTopic.to("topic3")
builder
}
def run() = {
val kafkaStreams = createStreams(topologyPlan())
kafkaStreams.start()
val kafkaConsumer = createConsumer()
val kafkaProducer = createProducer()
kafkaConsumer.subscribe(List("topic1").asJava)
while (true) {
val record = kafkaConsumer.poll(Duration.ofSeconds(5)).asScala
for (data <- record.iterator) {
kafkaProducer.send(new ProducerRecord[String, String]("topic2", data.value()))
}
}
}
Looking at the run method, the first two lines set up a streams object to that uses the topologyPlan() to listen for messages in 'topic2' and forward then to 'topic3'.查看 run 方法,前两行设置了一个流对象,它使用 topologyPlan() 来侦听“topic2”中的消息,然后转发到“topic3”。
The remaining lines show how a consumer can listen to a 'topic1' and use a producer to send them onward to 'topic2'.其余几行显示消费者如何收听“topic1”并使用生产者将它们发送到“topic2”。
The final point of the example here is Kafka is flexible enough to let you mix options depending on what you need, so the code above will take messages in 'topic1', and send them to 'topic3' via 'topic2'.这里示例的最后一点是 Kafka 足够灵活,可以让您根据需要混合选项,因此上面的代码将获取“topic1”中的消息,并通过“topic2”将它们发送到“topic3”。
If you want to see the code that sets up consumer, producer and streams, see the full class here .如果您想查看设置消费者、生产者和流的代码,请在此处查看完整类。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.