使用坚果时重复-> Elasticsearch Solution

Question

I have crawled some data using nutch and managed to inject it into elasticsearch. But I have one problem: If I inject the crawled data again it will create duplicates. Is there any way of disallowing this?

Has anyone managed to solve this or have any suggestions on how to solve it?

/Samus

Answer 1

One way , you can keep an index of check sum of all data you have entered into elasticSearch in some db and cross refer those before attempting to send data to elasticSearch. Or then you can run a "more like this" query to see similar documents and take decision based on that.

LINK - http://www.elasticsearch.org/guide/reference/query-dsl/mlt-field-query.html

Answer 2

If you index each page/document crawled with the same id in ElasticSearch it won't duplicate it. You could use a checksum/hash function to turn the page's URL into a distinct ID.

You can also use Operation_type to ensure that if that id is already indexed it should not reindex it:

The index operation also accepts an op_type that can be used to force a create operation, allowing for “put-if-absent” behavior. When create is used, the index operation will fail if a document by that id already exists in the index.

ElasticSearch index API