Logstash Elasticsearch compression

Question

I have a working ELK stack and would like to enable index compression.

The official store compression documentation tells me that I need to do it at index creation.

I couldn't find anything related to store compression or even index settings in the related logstash output documentation

Below is my logstash output configuration:

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

And the created index settings:

{
  "filebeat-2016.04.28": {
    "settings": {
      "index": {
        "creation_date": "1461915752875",
        "uuid": "co8bvXI7RFKFwB7oJqs8cA",
        "number_of_replicas": "1",
        "number_of_shards": "5",
        "version": {
          "created": "2030199"
        }
      }
    }
  }
}

Answer 1

You need to provide your own index template file in order to enable index compression.

So you need to create your filebeat-template.json file like this. This file will be used by logstash when creating a new filebeat index.

{
  "template" : "filebeat-*",
  "settings" : {
    "index.codec" : "best_compression"
  }
}

Then your elasticsearch output should be modified like this:

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    sniffing => true
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    template_name => "filebeat-template"
    template => "/path/to/filebeat-template.json"
  }
}

Then you can delete your existing filebeat-2016.04.28 index and relaunch logstash. The latter will create an index template called /_template/filebeat-template which will kick in everytime ES needs to create a new index whose name starts with filebeat- and it will apply the settings (among which the store compression one) present in the template.