Kafka connect 2.0.0 - 无法在集群模式下创建连接器 - 错误：请求领导重新配置连接器任务失败

Question

I have deployed Kafka connect (2.0.0) cluster across 2 nodes which is part of Hortonworks Data Platform 3.1.4. It gets deployed successfully and the kafka connect starts too. I am able to use REST commands to list the connectors using GET method.

Problem is, when I am trying to create the connector using POST method in cluster mode, it fails to create the connector or if it does create the connector then it fails to create the task. I see the following error in logs

    [2020-02-14 15:45:41,622] ERROR [pool-3-thread-1] [org.apache.kafka.connect.runtime.distributed.DistributedHerder] - Request to leader to reconfigure connector tasks failed
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Failed to start RestClient: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:74)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:383)
        at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:143)
        at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:91)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:273)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:71)
        ... 6 more
[2020-02-14 15:45:41,622] ERROR [pool-3-thread-1] [org.apache.kafka.connect.runtime.distributed.DistributedHerder] - Failed to reconfigure connector's tasks, retrying after backoff:
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Failed to start RestClient: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:74)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:383)
        at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:143)
        at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:91)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:273)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:71)
        ... 6 more

After some trail and errors it seems like when the task is created on the leader connect node it works and if it tries to create the task on non-leader connect node then it throws the above error. I am not very sure if my hypothesis is correct though.

Next when I force the connect cluster into standalone connect node by killing processes on other cluster nodes, the connectors and tasks get created work without any problems.

Can someone please throw some light on this issue, because currently I am unable to use Kafka connect in cluster mode.

Appreciate any ideas and inputs.

Answer 1

After some trail and errors it seems like when the task is created on the leader connect node it works and if it tries to create the task on non-leader connect node then it throws the above error.

I am not very sure if my hypothesis is correct though.

It is.

You need to adjust the rest.advertised.listener of the Connect workers such that they know about one another (advertise an IP or a DNS name within the network) in addition to listeners=HTTP://0.0.0.0:8083

https://kafka.apache.org/documentation/#connectconfigs