how to add bytes, session and source parameter in kibana to visualise suricata logs?

Question

I redirected all the logs(suricata logs here) to logstash using rsyslog. I used template for rsyslog as below:

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

for every incoming message, rsyslog will interpolate log properties into a JSON formatted message, and forward it to Logstash, listening on port 10514. Reference link: https://devconnected.com/monitoring-linux-logs-with-kibana-and-rsyslog/

(I have also configured logstash as mention on the above reference link)

I am getting all the column in Kibana discover( as mentioned in json-template of rsyslog) but I also require bytes, session and source column in kibana which I am not getting here. I have attached the snapshot of the column I am getting on Kibana here

Available fields(or say column) on Kibana are:

 @timestamp
t @version
t _type
t facility
t host
t message
t procid
t programname
t sysloghost
t _type
t _id
t _index
# _score
t severity

Please let me know how to add bytes, session and source in the available fields of Kibana. I require these parameters for further drill down in Kibana.

EDIT: I have added how my "/var/log/suricata/eve.json" looks like (which I need to visualize in Kibana. )

For bytes, I will use (bytes_toserver+bytes_toclient) which is an available inside flow. Session I need to calculate. Source_IP I will use as the source.

{"timestamp":"2020-05 04T14:16:55.000200+0530","flow_id":133378948976827,"event_type":"flow","src_ip":"0000:0000:0000:0000:0000:0000:0000:0000","dest_ip":"ff02:0000:0000:0000:0000:0001:ffe0:13f4","proto":"IPv6-ICMP","icmp_type":135,"icmp_code":0,"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":87,"bytes_toclient":0,"start":"2020-05-04T14:16:23.184507+0530","end":"2020-05-04T14:16:23.184507+0530","age":0,"state":"new","reason":"timeout","alerted":false}}

Answer 1

Direct answer

Read the grok docs in detail.

Then head over to the grok debugger with some sample logs, to figure out expressions. (There's also a grok debugger built in to Kibana's devtools nowadays)

This list of grok patterns might come in handy, too.

A better way

Use Suricata's JSON log instead of the syslog format, and use Filebeat instead of rsyslog. Filebeat has a Suricata module out of the box.

Sidebar: Parsing JSON logs

In Logstash's filter config section:

filter {

  json {
    source => "message"
    # you probably don't need the "message" field if it parses OK
    #remove_field => "message"
  }

}

[Edit: added JSON parsing]