删除弹性搜索河插件中未更新的记录

Question

In elastic search river , if i deleted a record in mysql , its still showing in index . I have enabled auto-commit also . How make mysql and elastic search in sync and also how to make delta-imports in elastic ?

   {
"type" : "jdbc",
"jdbc" : {
    "driver" : "com.mysql.jdbc.Driver",
    "url" : "jdbc:mysql://localhost:3306/testrivet",
    "user" : "root",
    "password" : "Gemini*123",
    "sql" : [
                        { 
                             "statement" : "select *,empid as _id from empdata"

                        }
       ],
    "strategy" : "simple",
 "schedule" : "0 0-59 0-23 ? * *",
 "autocommit" : true,
 "metrics": {enabled:true}
    },
    "index" : {
   "autocommit":true
    }
}

Answer 1

Indeed, if a record is deleted from your database, there's no way your JDBC river will be able to retrieve it anymore in order to delete the corresponding record in ES.

An alternative is to "soft-delete" records from your database by setting a flag (ie a new boolean column). The flag would be true when the record is active and false when the record is deleted. That way when your import process runs, you'd get all records and based on that flag you know you have to delete the documents from Elasticsearch.

There are other ways but they involve adding another component to the mix, so if this would do the job I'd suggest doing like that.