如何使用 Kafka Elasticsearch Sink Connector 写入多个不同的 Elasticsearch 集群

Question

Is is possible to use a single Kafka instance with the Elasticsearch Sink Connector to write to separate Elasticsearch clusters with the same index? Documentation . The source data may be a backend database or an application. An example use-case is that one cluster may be used for real-time search and the other may be used for analytics.

If this is possible, how do I configure the sink connector? If not, I can think of a couple of options:

1. Use 2 Kafka instances, each pointing to a different Elasticsearch cluster. Either write to both, or write to one and copy from it to the other.
2. Use a single Kafka instance and write a stream processor which will write to both clusters.

Are there any others?

Answer 1

Yes you can do this. You can use a single Kafka cluster and single Kafka Connect worker. One connector can write to one Elasticsearch instance, and so if you have multiple destination Elasticsearch you need multiple connectors configured.

The usual way to run Kafka Connect is in "distributed" mode (even on a single instance), and then you submit one—or more—connector configurations via the REST API.

You don't need a Java client to use Kafka Connect - it's configuration only. The configuration, per connector, says where to get the data from (which Kafka topic(s)) and where to write it (which Elasticsearch instance).

To learn more about Kafka Connect see this talk , this short video , and this specific tutorial on Kafka Connect and Elasticsearch