在不提供 .jar 的情况下在远程集群上运行 Flink 作业

Question

I have the following problem: I want to create a Flink job in IntelliJ on a local machine and run it on a remote cluster which is located in a VM. I used the createRemoteEnvironment function but anonymous classes as well as lambda expressions require jar files (if I didn't miss anything). Is there a (or another) way to run a Flink job on remote cluster without providing jar files? The following code is a simple Flink job I wanted to run on the remote cluster (without a jar).

public static void main(String[] args) throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.createRemoteEnvironment("192.168.56.101", 6123);

    DataStream<Tuple2<String, Integer>> dataStream = env
        .socketTextStream("192.168.56.102", 8080)
        .flatMap((String sentence, Collector<Tuple2<String, Integer>> out) ->
            {
                for (String word: sentence.split(" ")) {
                    out.collect(new Tuple2<String, Integer>(word, 1));
                }
            });


    // Alternative approach with an anonymous class
    /*DataStream<Tuple2<String, Integer>> dataStream = env
        .socketTextStream("localhost", 8080)
        .flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
        @Override
        public void flatMap(String sentence, Collector<Tuple2<String, Integer>> out) 
        throws Exception {
            for (String word: sentence.split(" ")) {
                out.collect(new Tuple2<String, Integer>(word, 1));
            }
        }
    });*/

    dataStream.print();

    env.execute("Window WordCount");
}

Thanks for any help!

Answer 1

Based on the description here the error may be less than intuitive, but it essentially means you need a dependency.

Conceptually, that means you need to provide the dependency.

And this is normally done by providing it in a jar.

So, based on how dependencies work in general I think the answer is that you must indeed provide the Jar to run the code that requires the dependency.