Flink State Schema Migration

Question

I have a flink streaming application on a standalone cluster that uses MemoryStateBackend. Kryo's TaggedFieldSerializer is being used as the default serialiser.

When I changed the state's schema and re-deployed the application, I got the following exception

Caused by: org.apache.flink.util.StateMigrationException: State migration isn't supported, yet.
at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.tryRegisterStateTable(HeapKeyedStateBackend.java:209)
at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.tryRegisterStateTable(HeapKeyedStateBackend.java:142)
at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend.createValueState(HeapKeyedStateBackend.java:234)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend$1.createValueState(AbstractKeyedStateBackend.java:315)
at org.apache.flink.api.common.state.ValueStateDescriptor.bind(ValueStateDescriptor.java:128)
at org.apache.flink.api.common.state.ValueStateDescriptor.bind(ValueStateDescriptor.java:35)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:312)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getPartitionedState(AbstractKeyedStateBackend.java:392)
at org.apache.flink.runtime.state.DefaultKeyedStateStore.getPartitionedState(DefaultKeyedStateStore.java:124)
at org.apache.flink.runtime.state.DefaultKeyedStateStore.getState(DefaultKeyedStateStore.java:60)

It will be really helpful if someone suggests me a way around it or should I use FsStateBackend to solve this problem.

PS If I wanted to use FsStateBackend on S3 for my flink application running on standalone cluster, what config changes has to be done.

Answer 1

Using FsStateBackend won't solve this issue, because it also uses HeapKeyedStateBackend under the hood, which is what's throwing this exception.

There is FLIP-22 to help address this issue of state migration, but it's not implemented yet.

Currently the best option I've heard of is to use an Avro-based serializer, as that can be implemented such that it handles old and new schemas seamlessly. But that's not something for the faint of heart.

Regarding FsStateBackend configuration, please see the (well written) documentation here .