How to debug flink streaming job in Intellij IDEA

Question

I want to create a Flink Job with some logic (Scala) and found several way to start Flink in the documentation. For example stand alone mode:

./bin/flink run ./examples/streaming/TopSpeedWindowing.jar

Could it be to start and debug a Flink job from Intellij Idea like a simple Spark Job with a main class?

  val spark = SparkSession.builder()
    .appName("Agg")
    .master("local")
    .getOrCreate()

  def streamCount() = {
    val lines: DataFrame = spark.readStream
...

  def main(args: Array[String]): Unit = {
    streamCount()
  }

What are the ways to do it with Flink? May be some specials libraries for unit tests or test containers.

Answer 1

Yes, you can certainly do that with Flink.

There are some examples you can easily run in your IDE in https://github.com/apache/flink-training -- in both Java and Scala. The training lessons that go with these exercises are part of the flink documentation: https://ci.apache.org/projects/flink/flink-docs-stable/learn-flink/ .