Kafka Streams仅提交KGroupedTable的最新消息

Question

I've got Kafka Streams application as follows:

static KafkaStreams build(AppConfig appConfig, SerdesHelper serdes) {
  final KStreamBuilder builder = new KStreamBuilder();

  builder
      .table(serdes.sourceKeySerde, serdes.sourceValueSerde, appConfig.sourceTopic)
      .groupBy(StreamBuilder::groupByMapper, serdes.intSerde, serdes.longSerde)
      .aggregate(
          StreamBuilder::initialize,
          StreamBuilder::add,
          StreamBuilder::subtract,
          serdes.sinkValueSerde)
      .to(serdes.intSerde, serdes.sinkValueSerde, appConfig.sinkTopic);

  return new KafkaStreams(builder, appConfig.streamConfig);
}

My concrete example groups records as follows

((k, v)) -> ((k), v[])

And while running this with dummy data of 3.000.000 messages having only two unique keys, I ended up having about 10.000 messages in sinkTopic in less than a minute and I hoped to get either 4/6 (based on the moment I manage to stop the application).

How can I ensure that only the key with the latest grouped value will be committed back to Kafka instead of every intermediate message?

Answer 1

It's stream processing, not batch processing. There is no "latest grouped value" -- the input is infinite, and thus, the output is infinite...

You can only reduce the number of intermediates by

1. increasing the KTable cache size (but this seems not to be an issue for your case as you have only 2 unique keys and thus both fit into the cache if you did not disable caching or
2. increasing the commit interval