filebeat-index-template.json for ElasticSearch 6.2.4

Question

I am running ElasticSearch 6.2.4. I tried to create Filebeat index template, but got the following error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "No handler for type [string] declared on field [message]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_default_]: No handler for type [string] declared on field [message]",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "No handler for type [string] declared on field [message]"
    }
  },
  "status" : 400
}

filebeat-index.template.json

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "string",
          "index": "analyzed"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "location" : { "type" : "geo_point" }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

I wonder if there is official filebeat-index-template.json that work for ElasticSearch 6.2.4

Other thing that I have tried

• Try filebeat -c "/etc/filebeat/filebeat.yml" export template > filebeat.template.json , but the filebeat will keep running forever without creating anything.
• I've tried to change "type": "string" to "type": "text", , but got another error where _all is deprecated.
• I've also tried to remove _all , but ElasticSearch keep have parsing error when Logstash send data to ElasticSearch.

Filebeat Version [Old]

I also try to find out the version of my Filebeat. I tried

> filebeat -v
Loading config file error: Failed to read /root/filebeat.yml: open /root/filebeat.yml: no such file or directory. Exiting.

> filebeat -v -c "/etc/filebeat/filebeat.yml"
(it struck forever) 

I am following this https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04 , but instead of using ElasticSearch 2.0 and Kibana 4.5, I am installing ElasticSearch 6.2.4, Kibana 6.2.4, and Logstash 6.2.4 and Ubuntu 16.04.4 LTS

Upgrading to Filebeat 6.2.4

Now I am upgrading Filebeat to 6.2.4. Now I get this error

Exiting: Could not start registrar: Error loading state: Error decoding states: json: cannot unmarshal object into Go value of type []file.State

I removed this error by rm /var/lib/filebeat/registry . Now I can do filebeat export template > template.json and it work fine now. I will close the question soon.

Answer 1

Try to use this elastic 6.0 modified json for filebeat-index.template.json

{
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "false",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "index": "true"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip": {
          "type": "object",
          "dynamic": true,
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

Basically I changed the message type from string to text . Also from elastic 6.0 onwards the index field uses true or false , instead of analyzed .

After running this command (as suggested in the blog you are referring to above):

curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json -H 'Content-Type: application/json'

I managed to get the correct confirmation from elastic:

{ 
  "acknowledged" : true
}

I haven't tested it yet, but please let me know if it works for you.

You will probably notice that the _all template is also removed from the original json. Why? Apparently it was depreciated in elastic 6.0 and there are ways to use copy_to instead as suggested in here but I haven't figured it out yet.

Answer 2

You should be able to use --es.version 6.2.4 when you generate the template to have it output the appropriate mappings for your version of elasticsearch.

Check out the instructions for Load the template manually (alternate method) . They show the following example for windows but it may work in linux too.

PS > .\filebeat.exe export template --es.version 6.6.2 | Out-File -Encoding UTF8 filebeat.template.json