How to read json file using filebeat and send it to elasticsearch via logstash

Question

This is my json log file. I'm trying to store the file to my elastic-Search through my logstash.

{"message":"IM: Orchestration","level":"info"}
{"message":"Investment Management","level":"info"}

Here is my filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - D:/Development_Avecto/test-log/tn-logs/im.log
  json.keys_under_root: true
  json.add_error_key: true
processors:
 - decode_json_fields:
     fields: ["message"]
output.logstash:
  hosts: ["localhost:5044"]

input{
    beats {
        port => "5044"
    }
}

filter {
    json {
      source => "message"
    }   
}   

output{
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "data"
    }
}

No able to view out put in elasticserach. Not able to find whats the error. filebeat log

2019-06-18T11:30:03.448+0530 INFO registrar/registrar.go:134 Loading registrar data from D:\\Development_Avecto\\filebeat-6.6.2-windows-x86_64\\data\\registry 2019-06-18T11:30:03.448+0530 INFO registrar/registrar.go:141 States Loaded from registrar: 10 2019-06-18T11:30:03.448+0530 WARN beater/filebeat.go:367 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning. 2019-06-18T11:30:03.448+0530 INFO crawler/crawler.go:72 Loading Inputs: 1 2019-06-18T11:30:03.448+0530 INFO log/input.go:138 Configured paths: [D:\\Development_Avecto\\test-log\\tn-logs\\im.log] 2019-06-18T11:30:03.448+0530 INFO input/input.go:114 Starting input of type: log; ID: 16965758110699470044 2019-06-18T11:30:03.449+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1 2019-06-18T11:30:34.842+0530 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312,"time":{"ms":312}},"total":{"ticks":390,"time":{"ms":390},"value":390},"user":{"ticks":78,"time":{"ms":78}}},"handles":{"open":213},"info":{"ephemeral_id":"66983518-39e6-461c-886d-a1f99da6631d","uptime":{"ms":30522}},"memstats":{"gc_next":4194304,"memory_alloc":2963720,"memory_total":4359488,"rss":22421504}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":10,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}} 2

Answer 1

https://www.elastic.co/guide/en/ecs-logging/dotnet/master/setup.html

Check step 3 at the bottom of the page for the config you need to put in your filebeat.yaml file:

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true