Kafka 连接到 ElasticSearch - LOG4J2 或连接被拒绝错误

Question

I am trying to configure Kafka Connect with Elastic search. But when I try to start it I am getting following ERRORs:

ERROR StatusLogger Log4j2 could not find a logging implementation.

ERROR Failed to create client to verify connection.  (io.confluent.connect.elasticsearch.Validator:120)
ElasticsearchException[java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused]; nested: ExecutionException[java.net.ConnectException: Connection refused]; nested: ConnectException[Connection refused]

I am trying to start it with following command in terminal:

connect-standalone.sh config/connect-standalone.properties config/elasticsearch.properties 

This is connect-standalone.properties:

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.flush.interval.ms=10000

# EDIT BELOW IF NEEDED
bootstrap.servers=localhost:9092
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/home/stjepan/kafka_2.13-3.2.3/connectors

This is elasticsearch.properties:

name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=wikimedia.recentchange
key.ignore=true
connection.url=localhost:9200
# connection.url=https://kafka-course-5842482143.eu-west-1.bonsaisearch.net
# connection.username=he6de7ka5o
# connection.password=yozz8ryqmg
type.name=kafka-connect

# necessary for this connector
schema.ignore=true
behavior.on.malformed.documents=IGNORE

# OVERRIDE
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

What am I doing wrong? Wikimedia Connector works fine... I didn't try anything else because I don't find any idea what can I change except ports, but it didn't helped...

Answer 1

You're probably referring to the course material of https://www.conduktor.io/apache-kafka-for-beginners (or the related course on Packt Pub)

I worked around this issue by using an OpenSearch Kafka Sink Connector:

https://github.com/aiven/opensearch-connector-for-apache-kafka

under https://github.com/aiven/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-elasticsearch.properties there is a Connector Config that you can merge into your existing config.

Hope that helps