logstash grok filter ignore certain parts of message

Question

I have a drupal watchdog log file that starts with syslog things like timestamp etc, and then has a pipe delimited number of things that I logged in watchdog. Now I am writing a grok filter rule to get fields out of that.

I have a few URLs in the message body so I use %{URI:request} for instance to get those. however that creates a field named port that is always null and I don't want to throw a lot of empty fields into my elasticsearch DB so I was wodnering how I could get rid of that empty array that looks like this : "port": [null, null, null] .

Answer 1

If you want to unconditionally drop a field, just add a remove_field => ['port'] to you grok block.

If you want to conditionally remove something, you can use either a ruby filter to check what is in the field before removing it, or use an if around a mutate { remove_field => ['port'] } filter.