简体繁体 English

如果 state 商店没有刷新到变更日志中，Kafka Streams 会重播源主题吗？

[英]Do Kafka Streams replay source topics if state store didn't get flushed into changelog?

原文 2022-11-13 18:52:50 2 1 apache-kafka/ apache-kafka-streams/ rocksdb

Question about State Stores and Rebalancing Let say:关于 State Stores and Rebalancing 的问题假设：

We have 2 input topics and we are aggregating records and pushing into a table.我们有 2 个输入主题，我们正在聚合记录并推入表中。
We have offset 10 of Topic 1 and offset 100 of Topic 2 are successfully aggregated我们已经成功聚合了主题 1 的偏移量 10 和主题 2 的偏移量 100
Problem: Offset 11-13 of Topic 1 and offset 101-102 of topic 2 aggregation happened in-memory but didn't make in changelog topic of state store.问题：主题 1 的偏移量 11-13 和主题 2 聚合的偏移量 101-102 发生在内存中，但没有出现在 state 存储的更改日志主题中。
Question: In changelog topic it has its own offset and new consumer will have state up to that.问题：在变更日志主题中，它有自己的偏移量，新消费者将拥有 state 到那个。 Will it have any link that Offset 11-13 and offset 101-102 has to be replayed?是否有必须重播偏移量 11-13 和偏移量 101-102 的任何链接？ Because these offsets might have been committed if I understand correctly?因为如果我理解正确的话，这些偏移量可能已经被提交了？

Please guide any material specific to this OR if you know the answer it will be much appreciated.请指导任何特定于此的材料，或者如果您知道答案，我们将不胜感激。 TIA TIA

I am aggregating and thinking if used cache (RocksDb) how it will work if something failed as highlighted above.我正在汇总并思考如果使用缓存（RocksDb），如果出现上面突出显示的失败，它将如何工作。

1 个解决方案

If i understand your question ², the changelog is source of truth for your state stores so in your case, you will have a data loss if your changelog is behind your local state store.如果我理解您的问题²，变更日志是您的 state 商店的真实来源，因此在您的情况下，如果您的变更日志落后于本地 state 商店，您将丢失数据。