Can a Kafka Streams State Store be entirely transient

Question

I am using a state store to track run time statistics about the messages passing through the stream. If a task fails/is migrated, we do not require the state to be re-created - we are more than happy for the state to start from 'fresh' once the task has migrated/re started. My understanding is if we don't enable change logging on the store then streams will have to re process the underlying topics to bring the state store up to date - this will cause a significant delay, and we do not want the overhead of the change log topic in the cluster. Put simply - can we have a 'throw away state' without any form of persistence?

Answer 1

You should be able to set acceptable.recovery.lag to INT.MAX . This should make any state store active even if it has not caught up the end offset. This won't start "fresh" exactly as it will catch the store up as it goes but there should be no waiting for the store to repopulate.

You can also set the change log topic retention time to 0. something like:

Properties streamsSettings = new Properties();
// Override default for both changelog and repartition topics
streamsSettings.put("topic.PARAMETER_NAME", "topic-value");
// alternatively, you can use
streamsSettings.put(StreamsConfig.topicPrefix("PARAMETER_NAME"), "topic-value");

between these I think you can do this