How to manage Flink application when Kafka broker is unavailable?

Question

I have a Flink application running in production which writes data to a Kafka topic owned by an external vendor.

We were notified by the vendor that they would be migrating their cluster and hence there will be downtime where the Kafka brokers will not be available.

My question is, what will happen to the Flink application data when the topic is not available to write data into? Can I allow my Flink application to continue running or should I stop it and wait for the brokers to be up and running?

Answer 1

The task will fail if it can't connect to the Kafka Sink. What it does after failing will depend on your Task Failure Recovery strategy.

If you don't want to keep an eye on when Kafka will be available again, a fixed-delay with infinite retries and a long delay or an exponential-delay strategy may be your best option to not overload your infrastructure too much with unnecessary restarts.