Custom log with Logstash+ logstash forwarder+ kibana+Elasticsearch

Question

I have configured Logstash+ logstash forwarder + kibana + Elasticsearch. Its working with system logs like Apache error logs & syslogs. Now, I have a application log, Like below,

Sender: sender@domain.com create_time: Thu Sep 4 02:41:54 2014 recipient: automation@domain.com

How can i read the above log file through kibana. Kindly someone help me in this regard. Your help much appropriated.

Answer 1

Your logstash should analyze your logs before to push them into elasticsearch.

I think that currently, your logstash has an input (lumberjack) and an output (ES). You should add at least one grok filter (the most popular) to extract new semantic fields.

Then you will be able to create visualizations according to these new fields :)

See here: logstash.net/docs/1.4.2/filters/grok

Answer 2

Use logstash forwarder to export logs and send the logs to logstash [ via lumberjack protocol].

In logstash filter plugin, use grok pattern for apache logs to get this working and output this elasticsearch node.

You can then use kibana to read the data from elasticsearch and visualise it accordingly.

Now, this can be replaced by using filebeat at the source end which can export the data and requires no development in logstash for something like apache logs. This will enable you to send data from filebeat [ apache logs] directly to elasticsearch and visualise from kibana.