Elasticsearch 5 connector in Apache Flink 1.4

Question

I'm a newbie to Flink, I'm trying to play with ElasticSearch connector but without success. I installed ElasticSearch 5.5, Kibana, and Flink 1.4 on Docker.

Here is my code:

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_2.11</artifactId>
        <version>1.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>1.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.11</artifactId>
        <version>1.4.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-rabbitmq_2.11</artifactId>
        <version>1.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
        <version>1.4.0</version>
    </dependency>
</dependencies>

while java code is:

DataStream<String> consumer = .....

    consumer.print();

    Map<String, String> config = new HashMap<>();
    config.put("cluster.name", "docker-cluster");
    config.put("bulk.flush.max.actions", "1");

    List<InetSocketAddress> transports = new ArrayList<>();
    transports.add(new InetSocketAddress("127.0.0.1", 9300));

    consumer.addSink(new ElasticsearchSink<>(config, transports, new ElasticsearchSinkFunction<String>() {
        public IndexRequest createIndexRequest(String element) {
            Map<String, String> json = new HashMap<>();
            json.put("id", "2");
            return Requests.indexRequest()
                    .index("signinindex").type("signin").source(json);
        }

        public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
        }
    }));

    env.execute();

But when I try to submit a new job in IDE, it seems to stay waiting, so I'm wondering if there are some problems with ElasticSearch 5.5 which conflicts with Flink. Thanks a lot.

[UPDATE]: I made some tests with ES 5.1.2 and now when submitting my jar I receive the following exception (I tried to upgrade Flink to 1.4.1 with all connectors too):

    java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: Could not run the jar.
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:90)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not run the jar.
    ... 9 more
Caused by: java.lang.RuntimeException: Could not look up the main(String[]) method from the class ProtoSignInMap: org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkFunction
    at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:495)
    at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
    at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:147)
    at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:83)
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69)
    ... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkFunction
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:489)
    ... 12 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 18 more

Obviously the main method exists.

Answer 1

Flink currently supports Elasticsearch up to version 5.1.x. For more info see: https://issues.apache.org/jira/browse/FLINK-7386