Can filebeat convert log lines output to json without logstash in pipeline?

Question

We have standard log lines in our Spring Boot web applications (non json). We need to centralize our logging and ship them to an elastic search as json.

(I've heard the later versions can do some transformation)

Can Filebeat read the log lines and wrap them as a json ? i guess it could append some meta data aswell. no need to parse the log line.

expected output : {timestamp : "", beat: "", message: "the log line..."}

i have no code to show unfortunately.

Answer 1

filebeat supports several outputs including Elastic Search .

Config file filebeat.yml can look like this:

# filebeat options: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/../file.err.log

processors:
   - drop_fields:
      # Prevent fail of Logstash (https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-6.3.html#custom-template-non-versioned-indices)
      fields: ["host"]
   - dissect:
      # tokenizer syntax: https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html.
      tokenizer: "%{} %{} [%{}] {%{}} <%{level}> %{message}"
      field: "message"
      target_prefix: "spring boot"

fields:
  log_type: spring_boot

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD"

Answer 2

Well it seems to do it by default. this is my result when i tried it locally to read log lines. it wraps it exactly like i wanted.

{  
   "@timestamp":"2019-06-12T11:11:49.094Z",
   "@metadata":{  
      "beat":"filebeat",
      "type":"doc",
      "version":"6.2.4"
   },
   "message":"the log line...",
   "source":"/Users/myusername/tmp/hej.log",
   "offset":721,
   "prospector":{  
      "type":"log"
   },
   "beat":{  
      "name":"my-macbook.local",
      "hostname":"my-macbook.local",
      "version":"6.2.4"
   }
}