Clearing flink state across operators in an atomic fashion

Question

Currently I am clearing the state of my operators via processingTimeTimer s in my KeyedProcessFunction s. I would like to consolidate this state clearing mechanism and clear the state across the difference operators in an atomic fashion as the logic for clearing the state is not only very similar for the different operators but also I am worried that the state may go out of sync if due to some reason state in one operator gets cleared but not in the other as I check for certain conditions in the onTimer method before clearing the state and so just because the onTimer method got called it doesn't necessarily mean that the state will be cleared. So far I haven't found a way online to do this. Any ideas anyone?

Answer 1

To ensure scalability, Flink avoids offering operations that require coordination across the cluster. The only exceptions to this are in the area of coordinating checkpoints, and transactional (two-phase commit) sinks, since these are needed for robust exact-only guarantees.

Your question is a bit unclear -- when you speak of atomically clearing state in different operators, perhaps you are referring to the various parallel instances of a particular stateful operator, or perhaps this refers to all of the instances of more than one stateful operator. Assuming the former, another approach you could take would be to broadcast a signal to all of the instances of an operator, instructing them as to what they should do. But each operator instance would process this message independently, so it's not obvious that this alternative would have much benefit.