Kafka：如何使用（手動），操作，確認和將結果發送到新主題

Question

我目前正在使用kafka-streams庫。

我正在嘗試做的事情 ：（使用kafka-streams）我正在嘗試從主題中消費，操縱消息值，確認該消息並將結果傳輸到另一個主題（下面的代碼）

Properties properties = new Properties();
.
.
properties.put("enable.auto.commit", false);

StreamBuilder builder = new StreamBuilder();
KStream kStream = builder.stream("MyTopic");
KafkaStream kafkaStream = new KafkaStream(builder.build(), properties)

KafkaProducer<String, String> producer = new KafkaProducer<>(properties);

kStream.forEach(new ForeachAction<String, String>(){

 @Override
 public void apply(Strings arg, String value){
  //Just doing some simple data manipulation
  String myValue = value + new Date().toString();

  //Sending result to new topic
  producer.send(new ProducerRecord<String,String>("MyTopicWithTimeStamp", myValue)):
  // Problem (1) Here -> How do I acknowledge this from here manually

  // Problem (2) How should I properly handle/close my producer (if at all)
 }
});

kafkaStram.start();

我不知道該怎么辦：使用kafka-streams庫正確確認消息

Answer 1

您可以使用KStream＃to直接寫入Kafka輸出主題，而不用創建和維護自己的生產者實例，如下所示：

final Properties props = new Properties();
...
final StreamsBuilder builder = new StreamsBuilder();
final KStream<String, String> source = builder.stream("MyTopic");
source.mapValues(record -> record + new Date().toString()).to("MyTopicWithTimeStamp", 
    Produced.with(Serdes.String(), Serdes.String()));
...
final KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Answer 2

您可以使用@amethystic也建議直接通過kafka流API進行編寫，而不是使用手動方法。

source.mapValues(record -> record + new Date().toString()).to("MyTopicWithTimeStamp", 
    Produced.with(Serdes.String(), Serdes.String()));

關於確認，您可以在流配置中設置該屬性，以確保生產者在發布到主題時收到確認。

final Properties props = new Properties();
props .put(StreamsConfig.producerPrefix(ProducerConfig.ACKS_CONFIG), "all");

您可以參考流配置： https : //kafka.apache.org/21/documentation/streams/developer-guide/config-streams.html#acks