Kafka Elasticsearch Connector for bulk operations

Question

I am using the Elasticsearch Sink Connector for operations (index, update, delete) on single records.

Elasticsearch also has a /_bulk endpoint which can be used to create, update, index, or delete multiple records at once. Documentation here .

Does the Elasticsearch Sink Connector support these types of bulk operations? If so, what is the configuration I need, or is there any sample code I can review?

Answer 1

Internally the Elasticsearch sink connector creates a bulk processor that is used to send records in a batch. To control this processor you need to configure the following properties:

• batch.size : The number of records to process as a batch when writing to Elasticsearch.
• max.in.flight.requests : The maximum number of indexing requests that can be in-flight to Elasticsearch before blocking further requests.
• max.buffered.records : The maximum number of records each task will buffer before blocking acceptance of more records. This config can be used to limit the memory usage for each task.
• linger.ms : Records that arrive in between request transmissions are batched into a single bulk indexing request, based on the batch.size configuration. Normally this only occurs under load when records arrive faster than they can be sent out. However it may be desirable to reduce the number of requests even under light load and benefit from bulk indexing. This setting helps accomplish that - when a pending batch is not full, rather than immediately sending it out the task will wait up to the given delay to allow other records to be added so that they can be batched into a single request.
• flush.timeout.ms : The timeout in milliseconds to use for periodic flushing, and when waiting for buffer space to be made available by completed requests as records are added. If this timeout is exceeded the task will fail.