How is memory managed in distributed runtime in Apache Flink?

Question

We are building an Apache Flink based data stream processing application in Java 8. We need to maintain a state-full list of objects which characteristics are updated every ten seconds via a source stream.

By specs we must use, if possible, no distributed storage. So, my question is about Flink's memory manager: in a cluster configuration, does it replicate the memory used by a task-manager? Or is there any way to use a distributed in-memory solution with Flink?

Answer 1

Have a look at Flink state . This way you can store it in flink's state which will be integrated with internal mechanisms like checkpointing/savepointing etc.

If you need to query it externally from other services a queryable state can be a good addition.