How to process json in logstash

Question

I'm new to the ELK Stack so please forgive the basic questions. What I would like to achieve is to make the "message" field a json object, so that I can filter it in kibana. Currently a message looks like this:

{
  "_index": "logstash-2017.01.16",
  "_type": "logs",
  "_id": "AVmpCcSiEWIcXZql",
  "_score": null,
  "_source": {
    "@timestamp": "2017-01-16T20:48:26.688Z",
    "port": 50018,
    "@version": "1",
    "host": "xxx.xx.x.xx",
    "message": "{\"@timestamp\":\"2017-01-16T20:48:26.642Z\",\"message\":\"\",\"tags\":[\"routing\"],\"source\":\"81caea3a2960/node /usr/src/app/src\",\"level\":\"info\",\"name\":\"apiservice-app\",\"server\":\"apiservice-app\",\"application\":\"apiservice-app\",\"port\":8080,\"hostname\":\"81caea3a2960\",\"pid\":16,\"req\":{\"method\":\"GET\",\"url\":\"/twitter/local/lists\",\"headers\":{\"host\":\"xxx.xx.x.xx:8080\",\"connection\":\"close\"},\"body\":{},\"hostname\":\"xxx.xx.x.xx\",\"ip\":\"::ffff:xxx.xx.x.xx\",\"originalUrl\":\"/twitter/local/lists\",\"params\":{},\"path\":\"/twitter/local/lists\",\"query\":{}},\"src\":{\"file\":\"/usr/src/app/src/libs/logstash.js\",\"line\":68}"}",
    "tags": []
  },
  "fields": {
    "@timestamp": [
      1484599706688
    ]
  },
  "sort": [
    1484599706688
  ]
}

My logstash.conf file looks like this

input {
 tcp {
  port => 5000
 }
}


output {
 elasticsearch {
  hosts => "elasticsearch:9200"
 }
}

The logs are being produced by a buyan logger in a node.js app:

'use strict';

const bunyan = require('bunyan');
const bunyantcp = require('bunyan-logstash-tcp');

let log = bunyan.createLogger({
      src: true,
      name: app.get('host'),
      server: app.get('host'),
      application: app.get('host'),
      port: app.get('port'),
      tags: ['routing'],
      streams: [{
        level: 'info',
          type: "raw",
          stream: process.stdout
      },{
          level: 'info',
          type: "raw",
          stream: bunyantcp.createStream({
              host: serviceLogstashSettings.address,
              port: serviceLogstashSettings.port
          })
      }],
      level: 'debug'
});

What plugins do I need to install in logstash and how would my logstash.conf file need to look to change the "message" field into a json field?

Answer 1

That's a good start. Now all you need is to use a json codec in your tcp input to parse the incoming data as JSON, like this:

input {
 tcp {
  port => 5000
  codec => `json`
 }
}