Grok filtering in logstash for multiple defined patterns

Question

I am trying to filter my logs matching few patterns I have. eg:

E/vincinity/dholland_view_sql_global/IN/Cluster_Node/SSL-CACHE/Dsal1
F/vincinity/dholland_view_sql_local/IN/Cluster_Node3/SSL-CACHE/Dsal4

R/vincinity/dholland_view_sql_bran/IN/Cluster_Node/Sample/vr1.log

Now I want to grep these 3 paths from a bunch of logs: basically the pattern that I want to extract is logs containing "vincinity" "sql" and "IN" so with regex it would be simply *vincinity*sql*IN* I tried this grok filter:

grok {

    match => { "Vinc" => "%{URIPATHPARAM:*vincinity*sql*IN*}" }

  }

Then I get _grokparsefailure in kibana - I'm brand new to grok, so perhaps I'm not approaching this correctly.

Answer 1

From the grok filter documentation

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The way the grok filter should work is

grok {
  match => {
    "message" => "%{PATTERN:named_capture}"
  }
}

Where message is the field that you want to parse, this is the default field that most inputs place your unparsed loglines in.

The URIPATHPARAM pattern is one predefined in logstash through a regex language called Onigurama. It may match your whole log message, but it will not capture certain chunks of it for you.

For help constructing a grok pattern, check out the docs , they link to a couple useful pattern construction tools.

Answer 2

The correct format for using a custom pattern in your grok block is:

(?<field_name>the pattern here)

or you can define your own custom pattern (using regular expression) in seperate file (my-pattern.txt) like this :

MYPATH_MUST_BE_UPPERCASE Regex_Pattern

save it in ./patterns directory and then use it this way:

grok {
     patterns_dir => "./patterns"
     match => ["message" , "%{MYPATH_MUST_BE_UPPERCAS:path}"]
}

in your case :

(?<vincinity>(?>/\s*.*?vincinity.*?\s*)+)
(?<sql>(?>/\s*.*?sql.*?/\s*)+)
(?<in>(?>\s*.*?(IN).*?\s*)+)