How do I index regular expressions in Elastic Stack?

Question

Using ELK/Elastic Stack for log management, I want to build reports based on log lines like this:

2018-01-31 11:50:00.212 Loading the user images took 234 ms 
2018-01-31 10:23:01.984 Loading the user images took 331 ms 
2018-01-31 10:12:41.323 Loading the user images took 512 ms 

So, the report would take lines with the string Loading the user images took

Then the regex would extract the number as in /took (\\d+) ms/ and build a chart or notification.

How do I do this? Perhaps this involves defining an index somehow?

Answer 1

I would use logstash grok filters.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

Use this site to test your grok patterns user-friendly way:

http://grokconstructor.appspot.com/do/match

I made a pattern that extracts the ms to a timeTaken field:

%{GREEDYDATA}took %{NUMBER:timeTaken}

If you need more data, use the online page for testing and search for grok patterns. Visualization is done in Kibana.