The implementation of the provided ElasticsearchSinkFunction is not serializable(flink-connector-elasticsearch6_2.11)

问题

"non-serializable" error occurs when I follow flink document to write data via flink streaming. I use flink1.6,Elastic-Search-6.4 and flink-connector-elasticsearch6.
My code is like

@Test
  public void testStringInsert() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
    env.enableCheckpointing(100); //
    DataStreamSource<String> input = env.fromCollection(Collections.singleton("testData"));
    List<HttpHost> httpHosts = new ArrayList<>();
    httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));
    ElasticsearchSink.Builder<String> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<String> ()  {
          public IndexRequest createIndexRequest(String element) {
            Map<String, String> json = new HashMap<>();
            json.put("data", element);
            return Requests.indexRequest()
                .index("my-index")
                .type("my-type")
                .source(json);
          }
          @Override
          public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
          }
        }
    );
    esSinkBuilder.setBulkFlushMaxActions(1);
    input.addSink(esSinkBuilder.build());
    env.execute("test es string insert");
  }

When I run the code above,I got the exception

java.lang.IllegalArgumentException: The implementation of the provided ElasticsearchSinkFunction is not serializable. The object probably contains or references non-serializable fields.

	at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
	at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.<init>(ElasticsearchSinkBase.java:216)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:71)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:60)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink$Builder.build(ElasticsearchSink.java:208)
	at com.lianlianpay.erebus.erebusaccess.ElasticsearchSinkTest.testStringInsert(ElasticsearchSinkTest.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

I searched a lot and still puzzled.I did not pass Object to ElasticsearchSink,other than String,and String is serializable without a doubt.I really can not understand , what is wrong with my code or the develop enviroment ?

回答1:

I just met the same error message. the problem was with credentialsProvider which should be instantiated within the RestClientFactory

Example in Scala:

class SecuredRestClientFactory (username: String, password: String) extends RestClientFactory {
  override def configureRestClientBuilder(builder: RestClientBuilder): Unit = {
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
      override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
        val credentialsProvider = new BasicCredentialsProvider
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password))
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
      }
    })
  }}
esSinkBuilder.setRestClientFactory(new SecuredRestClientFactory(elasticUser,elasticPass))

回答2:

Flink Elasticsearch Connector 7

Please find a working and detailed answer which I have provided here, Which is written in Scala.

来源：https://stackoverflow.com/questions/52885310/the-implementation-of-the-provided-elasticsearchsinkfunction-is-not-serializable