Logstash output to ES gives me error code '400'

Question

Can I please ask for a help?

I am using following:

• logstash v.6.1.5
• AWS ES v.7.9

I have following logstash output:

    elasticsearch {
      hosts => ["es_host"]
      ssl => true
      user => "user"
      password => "password"
      template => '/etc/logstash/conf.d/template-default.json'
      template_overwrite => true
      index => "log-default-%{+yyyy-MM-dd}"
      ilm_enabled => false
    }

And this is the template:

    "order": 1,
    "index_patterns": [
        "log-default-*",
        "log-http-*"
    ],
    "settings": {
        "number_of_shards": 6,
        "number_of_replicas": 2
    },
    "mappings": {
        "_default_": {
            "dynamic_templates": [
                {
                    "ips": {
                        "match_pattern": "regex",
                        "match": "^(?:orig_)?(?:src|dst)ip$",
                        "mapping": {
                            "type": "ip"
                        }
                    }
                },
                {
                    "strings": {
                        "match": "*",
                        "unmatch": "*message",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "keyword"
                        }
                    }
                }
            ]
        }
    }
}

When I start logstash, I receive an error as per the following. Does anyone have an idea what could be wrong?

[ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://es-host:443/_template/logstash'

Answer 1

Your template-default.json file should contain this, ie remove the _default_ key and you're good to go.

{
  "order": 1,
  "index_patterns": [
    "log-default-*",
    "log-http-*"
  ],
  "settings": {
    "number_of_shards": 6,
    "number_of_replicas": 2
  },
  "mappings": {
    "dynamic_templates": [
      {
        "ips": {
          "match_pattern": "regex",
          "match": "^(?:orig_)?(?:src|dst)ip$",
          "mapping": {
            "type": "ip"
          }
        }
      },
      {
        "strings": {
          "match": "*",
          "unmatch": "*message",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}