使用Apache Kafka将数据从MSSQL同步到Elasticsearch

Question

I'm currently running a text search in SQL Server, which is becoming a bottleneck and I'd like to move things to Elasticsearch for obvious reasons, however I know that I have to denormalize data for best performance and scalability.

Currently, my text search includes some aggregation and joining multiple tables to get the final output. Tables, that are joined, aren't that big (up to 20GB per table) but are changed (inserted, updated, deleted) irregularly (two of them once in a week, other one on demand x times a day).

My plan would be to use Apache Kafka together with Kafka Connect in order to read CDC from my SQL Server, join this data in Kafka and persist it in Elasticsearch, however I cannot find any material telling me how deletes would be handled when data is being persisted to Elasticsearch.

Is this even supported by the default driver? If not, what are the possibilities? Apache Spark, Logstash?

Answer 1

I am not sure whether this is already possible in Kafka Connect now, but it seems that this can be resolved with Nifi.

Hopefully I understand the need, here is the documentation for deleting Elasticsearch records with one of the standard NiFi processors:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-elasticsearch-5-nar/1.5.0/org.apache.nifi.processors.elasticsearch.DeleteElasticsearch5/