How to store a part of log into a file using LogStash

Question

I'm processing a log file with the help of logstash aggregate filter with grok having multiple patterns.

Now while processing the logs I want to extract a part of the log with some regex and store it into a file. For example, let's say my log is:

id:0422  time:[2013-11-19 02:34:58]   level:INFO  text:(Lorem Ipsum is simply dummy text of the printing and typesetting industry) 

In this log the text will be different at every time. I have a regex with help of it I can match a part of text that can occure in logstash So if I find something in that text with help of that regex while logstash indexing into elastic I want to store it into some file or something

Is it possible to achieve this?

Answer 1

There are different solutions for this:

• create a filter using ruby code that will be triggered to write in a specific format when you have all the event data together
• create a separate output which will be triggered based on an if statement to a file, this will be the preferred way of working as it is clear that it is an output.

Depending on the fact if you want to send all data or not, or have it look different or not you might need to use the clone function in order to clone the event into two different ones which can be manipulated apart from each other using tags.