Elasticsearch with node.js how to make unique field when insert document to an index

Question

I'm developing search engine to my project and i'm using Elasticsearch and node.js for the server.

Every night I have a parser that scrap data from some website and insert it to the db. For now it duplicates the data that I already have.

Can I make a unique field inside the index when insert a document for example title : {unique : true} and by that it will not insert me a document that come with this title

Here is my code :

async function insertManual(manual) {
  return new Promise(async (resolve, reject) => {
    const result = await client.index({
        index : 'completeindexthree',
        body : {
            brand : manual.brand,
            category : manual.category,
            url : manual.url,
            title : manual.title, // example {unique : true}
            parsingData : new Date().toString()
        }
    })
    await client.indices.refresh({index: 'completeindexthree'})
    resolve(result);
  })
} 

the second is , how can i delete all my duplicates that already got in by title from that index in node.js not from logstach ?

Answer 1

Tldr;

Yes it is possible, not by using the unique keyword though. According to the documentation , if you set an _id and this id exist already it will be replaced/overwrite

If the target is an index and the document already exists, the request updates the document and increments its version.

Furthermore you will find this section

Using _create guarantees that the document is only indexed if it does not already exist.

To fix

You should set an _id per document and use the create

Your code may look like the following:

async function insertManual(manual) {
  return new Promise(async (resolve, reject) => {
    const result = await client.create({
        index : 'completeindexthree',
        id: manual.id,   // <- Here is your unique id.
        body : {
            brand : manual.brand,
            category : manual.category,
            url : manual.url,
            title : manual.title, // example {unique : true}
            parsingData : new Date().toString()
        }
    })
    await client.indices.refresh({index: 'completeindexthree'})
    resolve(result);
  })
} 

Answer 2

If you don't give an id, elastic search creates a unique id, if you do, it creates the id you gave.

payload should be like this

{
 id:"you_unique_id",
 body:{foo,"bar"}

}