Elasticsearch Reindex

Question

Theres an index that I want to apply updated mappings to, I have done my best to follow the documentation on ES and Stackoverflow but I am now stuck.

The original index: logstash-index-YYYY.MM with data in it
I created index: logstash-index-new-YYYY.MM (which has a template for the new mapping)

Using the following query:
/logstash-index-YYYY.MM/_search?search_type=scan&scroll=1m
{ "query": { "match_all": {} }, "size": 30000 }

I get a _scroll_id and I have less than 30k docs so I should only need to run once.

How do I use that id to push the data into the new index?

Answer 1

You are not using scrollid to push the data into the new index. You use it to get another portion of data from the scroll query.

When you run scan query, first pass doesn't return any results, it scans through shards in your cluster and returns scrollid. Another pass (using scrollid from first one) will return actual results.

If you want to put that data into new index you should write some kind of simple program in language of your choice that will get this data and then put it into your new index.

There is a very good article on elasticsearch blog how to change mappings of your indices on the fly. Unfortunately, reindexing itself is not covered there.