Kafka streams state store distribution

Question

I have a kafka application that runs on multiple instances and I want to use state store for caching a few data fields. In case of multiple application instances if one instance goes down, does the local state store of one instance gets copied to other instance? What happens when the instance comes back? How are the state stores connected to the data keys for proper redistribution?

Answer 1

if one instance goes down, does the local state store of one instance gets copied to other instance?

If you don't have a standby replica, then the task will read the changelog topic from the beginning to rebuild the store, effectively making a copy, yes.

In the docs ,

Starting in 2.6, Kafka Streams will guarantee that a task is only ever assigned to an instance with a fully caught-up local copy of the state, if such an instance exists. Standby tasks will increase the likelihood that a caught-up instance exists in the case of a failure

---

How are the state stores connected to the data keys for proper redistribution?

Partitions are mapped to task threads (refer same page).