运行多个 kafka 独立 hdfs 连接器时出错

Question

We are trying to launch multiple standalone kafka hdfs connectors on a given node.

For each connector, we are setting the rest.port and offset.storage.file.filename to different ports and path respectively.

Also kafka broker JMX port is @ 9999.

When I start the kafka standalone connector, I get the error

Error: Exception thrown by the agent: java.rmi.server.ExportException: Port already in use: 9999; nested exception is: java.net.BindException: Address already in use (Bind failed)

Though the rest.port is set to 9100

kafka version: 2.12-0.10.2.1

kafka-connect-hdfs version: 3.2.1

Please help.

Answer 1

We are trying to launch multiple standalone kafka hdfs connectors on a given node.

Have you considered running these multiple connectors within a single instance of Kafka Connect? This might make things easier.

Kafka Connect itself can handle running multiple connectors within a single worker process. Kafka Connect in distributed mode can run on a single node, or across multiple ones.

Answer 2

For those who trying to use rest.port flag and still getting Address already in use error. That flag has been marked as deprecated in KIP-208 and finally removed in PR .
From that point listeners can be used to change default REST port.
Examples from Javadoc

listeners=HTTP://myhost:8083
listeners=HTTP://:8083

Configuring and Running Workers - Standalone mode

Answer 3

You may have open Kafka Connect connections that you don't know about. You can check this with:

ps -ef | grep connect

If you find any, kill those processes.