logstash json filter source

Question

I cannot get the message field to decode from my json log line when receiving via filebeat.

Here is the line in my logs:

{"levelname": "WARNING", "asctime": "2016-07-01 18:06:37", "message": "One or more gateways are offline", "name": "ep.management.commands.import", "funcName": "check_gateway_online", "lineno": 103, "process": 44551, "processName": "MainProcess", "thread": 140735198597120, "threadName": "MainThread", "server": "default"}

Here the logstash config. I tried with and without the codec . The only difference is that the message is being escaped when I use the codec.

 input {
  beats {
    port => 5044   
    codec => "json"
  }
}

filter {
 json{
 source => "message"
 }
}

Here is the json as it arrives in elasticsearch:

    {
  "_index": "filebeat-2016.07.01",
  "_type": "json",
  "_id": "AVWnpK519vJkh3Ry-Q9B",
  "_score": null,
  "_source": {
    "@timestamp": "2016-07-01T18:07:13.522Z",
    "beat": {
      "hostname": "59b378d40b2e",
      "name": "59b378d40b2e"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "message": "{\"levelname\": \"WARNING\", \"asctime\": \"2016-07-01 18:07:12\", \"message\": \"One or more gateways are offline on server default\", \"name\": \"ep.controllers.secure_client\", \"funcName\": \"check_gateways_online\", \"lineno\": 80, \"process\": 44675, \"processName\": \"MainProcess\", \"thread\": 140735198597120, \"threadName\": \"MainThread\"}",
    "offset": 251189,
    "source": "/mnt/ep_logs/ep_.json",
    "type": "json"
  },
  "fields": {
    "@timestamp": [
      1467396433522
    ]
  },
  "sort": [
    1467396433522
  ]
}

What I would like is that contents from the message object are decoded.

Many thanks

Answer 1

When that happens, it's usually because your Filebeat instance is configured to send documents directly to ES.

In your filebeat configuration file, make sure to comment out the elasticsearch output.