Reindexing a index in elasticsearch - ELK

Question

What's the best practise to reindex an elastic search index? This post has few steps which involves stopping logstash indexer before reindexing an index but that's not an option for me as its a production server.

I had a problem where there were no *.raw fields in an index, because of missing default mapping template. I used the mapping template from Elasticsearch found here and configured my ES cluster to use it, but I guess it will only be used when either a new index is created or when I explicitly reindex an existing index.

Also /_template?pretty resulted back an empty response, but after adding the aforementioned template /_template?pretty reveals a new template, but will the new index that would be created use this new template automatically? Or do I need to configure ES explicitly asking it to use them?

I would really appreciate some help here.

Answer 1

The best way to do this IMO is to have an alias pointing to your actual index. ( https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html ), something like my-index-alias pointing to my-actual-index-20170101

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "my-actual-index-20170101", "alias" : "my-index-alias" } }
    ]
}

When you query (or add stuffs) to your index, you always use the alias name.

If you need to reindex, you reindex to another index ( my-actual-index-20170127 by example) and when the reindexing is done, you update the alias to point to the new index.

POST /_aliases
{
    "actions" : [
        { "remove" : { "index" : "my-actual-index-20170101", "alias" : "my-index-alias" } },
        { "add" : { "index" : "my-actual-index-20170127", "alias" : "my-index-alias" } }
    ]
}

It'll allow you to reindex an index without changing any code / having any downtime.

Now, regarding the actual reindexing problem-- I've been using the elasticsearch-reindex node app with a lot of success ( https://www.npmjs.com/package/elasticsearch-reindex ).

Its usage is easy :

after installing it with npm ( npm install -g elasticsearch-reindex ), you can just run

elasticsearch-reindex -f http://localhost:9200/my-actual-index-20170101/{type of documents} -t http://localhost:9200/my-actual-index-20170127

Answer 2

A simple way to reindex your documents(using Kibana Sense) in another index with the new template applied can be:

POST /_reindex
{
  "source": {
    "index": "source-index"
  },
  "dest": {
    "index": "dest-index"
  }
}

Answer 3

Another way to do a lot of different actions with Elasticsearch including re-indexing is Curator .

Its latest version is compatible with ES 5.x and 6.x