快速将整个MongoDB集合索引到Elastcticsearch

Question

I have a collection in MongoDB which I am indexing into Elasticsearch. I am doing this in a C# process. The collection has 100 million documents, and for each document, I have to query other documents in order to denormalise into the Elasticsearch index.

This all takes time. Reading from MongoDB is the slow part (indexing is relatively quick). I am batching the data from MongoDB as efficiently as I can but the process takes over 2 days.

This only has to happen when the mapping in Elasticsearch changes, but that has happened a couple of times over the last month. Are there any ways of improving the performance for this?

Answer 1

Maybe you don't need launch import from scratch (I mean import from MongoDB), when you change mappings. Read this: Elasticsearch Reindex API

When you need to change mapping you must:

1. Create new index with new mapping
2. Reindex data from the old index into a new index using the built-in feature of elasticsearch.

After this old documents will be indexed with new mappings inside the new index. And built-in reindex in elasticsearch will work more quickly, than import from MongoDB via HTTP API.

If you will use reindex, don't forget to use parameter wait_for_completion (this parameter described in the documentation). This will run the reindex in the background.

Is this approach will solve your problem?