How to use BasicAuth with ElasticSearch Connector on Flink

Question

I want to use the elastic producer on flink but I have some trouble for authentification: I have Nginx in front of my elastic search cluster, and I use basic auth in nginx.

But with the elastic search connector I can't add the basic auth in my url (because of InetSocketAddress)

did you have an Idea to use elasticsearch connector with basic auth ?

Thanks for your time.

there is my code :

 val configur = new java.util.HashMap[String, String]

    configur.put("cluster.name", "cluster")

    configur.put("bulk.flush.max.actions", "1000")

    val transportAddresses = new java.util.ArrayList[InetSocketAddress]
    transportAddresses.add(new InetSocketAddress(InetAddress.getByName("cluster.com"), 9300))


    jsonOutput.filter(_.nonEmpty).addSink(new ElasticsearchSink(configur,
                                                                transportAddresses,
                                                                new ElasticsearchSinkFunction[String] {
      def createIndexRequest(element: String): IndexRequest = {

        val jsonMap = parse(element).values.asInstanceOf[java.util.HashMap[String, String]]

        return Requests.indexRequest()
          .index("flinkTest")
          .source(jsonMap);
      }

      override def process(element: String, ctx: RuntimeContext, indexer: RequestIndexer) {
        indexer.add(createIndexRequest(element))
      }
    }))

Answer 1

Flink uses the Elasticsearch Transport Client which connects using a binary protocol on port 9300. Your nginx proxy is sitting in front of the HTTP interface on port 9200.

Flink isn't going to use your proxy, so there's no need to provide authentication.

Answer 2

If you need to use a HTTP Client to connect Flink with Elasticsearch, one solution is to use Jest Library .

You have to create a custom SinkFunction, like this basic java class :

    package fr.gfi.keenai.streaming.io.sinks.elasticsearch5;

    import org.apache.flink.configuration.Configuration;
    import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;

    import io.searchbox.client.JestClient;
    import io.searchbox.client.JestClientFactory;
    import io.searchbox.client.config.HttpClientConfig;
    import io.searchbox.core.Index;

    public class ElasticsearchJestSinkFunction<T> extends RichSinkFunction<T> {

        private static final long serialVersionUID = -7831614642918134232L;

        private JestClient client;

        @Override
        public void invoke(T value) throws Exception {

            String document = convertToJsonDocument(value); 

            Index index = new Index.Builder(document).index("YOUR_INDEX_NAME").type("YOUR_DOCUMENT_TYPE").build();
            client.execute(index);

        }

        @Override
        public void open(Configuration parameters) throws Exception {

            // Construct a new Jest client according to configuration via factory
            JestClientFactory factory = new JestClientFactory();
            factory.setHttpClientConfig(new HttpClientConfig.Builder("http://localhost:9200")
                    .multiThreaded(true)
                    // Per default this implementation will create no more than 2 concurrent
                    // connections per given route
                    .defaultMaxTotalConnectionPerRoute(2)
                    // and no more 20 connections in total
                    .maxTotalConnection(20)
                    // Basic username and password authentication
                    .defaultCredentials("YOUR_USER", "YOUR_PASSWORD")
                    .build());
            client = factory.getObject();
        }

        private String convertToJsonDocument(T value) {
            //TODO
            return "{}";
        }

    }

Note that you can also use bulk operations for more speed.

An exemple of Jest implementation for Flink is described at the part "Connecting Flink to Amazon RS" of this post