The implementation of the provided ElasticsearchSinkFunction is not serializable(flink-connector-elasticsearch6_2.11)

Question

"non-serializable" error occurs when I follow flink document to write data via flink streaming. I use flink1.6,Elastic-Search-6.4 and flink-connector-elasticsearch6.
My code is like

 @Test public void testStringInsert() throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); env.enableCheckpointing(100); // DataStreamSource<String> input = env.fromCollection(Collections.singleton("testData")); List<HttpHost> httpHosts = new ArrayList<>(); httpHosts.add(new HttpHost("127.0.0.1", 9200, "http")); ElasticsearchSink.Builder<String> esSinkBuilder = new ElasticsearchSink.Builder<>( httpHosts, new ElasticsearchSinkFunction<String> () { public IndexRequest createIndexRequest(String element) { Map<String, String> json = new HashMap<>(); json.put("data", element); return Requests.indexRequest() .index("my-index") .type("my-type") .source(json); } @Override public void process(String element, RuntimeContext ctx, RequestIndexer indexer) { indexer.add(createIndexRequest(element)); } } ); esSinkBuilder.setBulkFlushMaxActions(1); input.addSink(esSinkBuilder.build()); env.execute("test es string insert"); }

When I run the code above,I got the exception

 java.lang.IllegalArgumentException: The implementation of the provided ElasticsearchSinkFunction is not serializable. The object probably contains or references non-serializable fields. at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.<init>(ElasticsearchSinkBase.java:216) at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:71) at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:60) at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink$Builder.build(ElasticsearchSink.java:208) at com.lianlianpay.erebus.erebusaccess.ElasticsearchSinkTest.testStringInsert(ElasticsearchSinkTest.java:151) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

I searched a lot and still puzzled.I did not pass Object to ElasticsearchSink,other than String,and String is serializable without a doubt.I really can not understand , what is wrong with my code or the develop enviroment ?

Answer 1

I just met the same error message. the problem was with credentialsProvider which should be instantiated within the RestClientFactory

Example in Scala:

class SecuredRestClientFactory (username: String, password: String) extends RestClientFactory {
  override def configureRestClientBuilder(builder: RestClientBuilder): Unit = {
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
      override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
        val credentialsProvider = new BasicCredentialsProvider
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password))
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
      }
    })
  }}
esSinkBuilder.setRestClientFactory(new SecuredRestClientFactory(elasticUser,elasticPass))

Answer 2

Flink Elasticsearch Connector 7

Please find a working and detailed answer which I have provided here , Which is written in Scala.