kafka 连接elasticsearch 连接器（接收器）数据移动（几乎）实时

Question

interested in moving data from kafka to elasticsearch. so i have setup kafka connect elasticsearch connector .

although i reviewed all the documentation and the configuration , i have yet found how to configure the connector to move data from kafka to elasticsearch in (near) real-time.

that is, making the connector consume message from the (kafka) topic each second and write them into elasticsearch, mimking streaming of messages from kafka to elasticsearch.

this is the current configuration for the connector:

{
  "name": "elasticsearch-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "foo",
    "key.ignore": "true",
    "schema.ignore": "true",
    "connection.url": "http://elasticsearch:9200",
    "type.name": "kafka-connect",
    "name": "elasticsearch-sink"
  }
}

how does the connector need to be configured so it'll "stream" message from kafka to elasticsearch?

Answer 1

Since Kafka Sink Connector is based on consumer , you can control polling interval via consumer properties (eg max.poll.interval.ms , max.poll.records ). To configure, just add consumer property with prefix consumer. to your connector config:

consumer.max.poll.records=1