How to keep flink batch job running continuously on local

Question

I am practicing file reading through the flink batch processing mechanism on a Windows 10 machine.

I downloaded flink-1.7.2-bin-hadoop24-scala_2.12.tgz from flink's official site and executed start-cluster.bat .

I uploaded the jar though Flink's UI and was able to execute the job but the job finished in a matter of seconds.

I want to keep the job running continuously so that I can test my use case .

Can you guide my possible ways to achieve this?

Answer 1

In Flink, batch jobs run until all of their input has been processed, at which point they have finished and are terminated. If you want continuous processing, then you should either

• use some deployment automation (outside of Flink) to arrange for new batch jobs to be created as needed, or
• implement a streaming job

In your case it sounds like you might be looking for the FileProcessingMode.PROCESS_CONTINUOUSLY option on StreamExecutionEnvironment.readfile -- see the docs for more info.