How to do an automated index creation at ElasticSearch?

Question

How to do an automated index creation at ElasticSearch?

Just like wordpress? See: http://gibrown.wordpress.com/2014/02/06/scaling-elasticsearch-part-2-indexing/

In our case we create one index for every 10 million blogs, with 25 shards per index.

Any light?

Thanks!

Answer 1

You do it in whatever your favorite scripting language is. You first run a query getting a count of the number of documents in the index. If it's beyond a certain amount you create a new one, either via an Elasticsearch API or a curl.

Here's the query to find the number of docs:

curl --XGET 'http://localhost:9200/youroldindex/_count'

Here's the index creation curl:

curl -XPUT 'http://localhost:9200/yournewindex/' -d '{
    "settings" : {
        "number_of_shards" : 25,
        "number_of_replicas" : 2
    }
}'

You will also probably want to create aliases so that your code can always point to a single index alias and then change the alias as you change your hot index:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html

You will probably want to predefine your mappings too:

curl -XPUT 'http://localhost:9200/yournewindex/yournewmapping/_mapping' -d '
{
    "document" : {
        "properties" : {
            "message" : {"type" : "string", "store" : true }
        }
    }
}
'

Elasticsearch has fairly complete documentation, a few good places to look:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html