Print a complete line if the regex pattern in fluentd is not matching

Question

I have a requirement to print a part of a message that matches a specific pattern. In some cases the message doesnt have that pattern; in that case i need to display the whole line. For eg the value in MESSAGE field can be any of the two:

Case 1: 2021-03-31 12:12:05.856 LOG: Message <checked [abc]>

Case 2: No Message was found

The fluentd filter is:

<filter docker>
  @type record_transformer
  enable_ruby true  
  <record>
   MESSAGE ${record["MESSAGE"].scan(/:\ (.*+)$/).first} 
  </record>  
</filter>

The filter works fine for case 1. and it prints Message <checked [abc]> but returns empty for case 2 wherein i need it to print No Message was found . How can i print the message even when condition is not satisfied. Thanks

Answer 1

You can do it like this:

MESSAGE ${record["MESSAGE"].scan(/^.+: (.+)$|^(.*)$/).first.compact}

Edit :

The last part of the regexp (in your case, the second half) will capture the line when the previous parts don't match. It's a common technique in regexps; some people may call it "garbage collection"?

As I introduced a second capture group, you'll get an array containing two groups. If a group didn't capture anything then its value will be nil . That's why you can call compact to get rid of the nil captures; that will reduce the array to only one element.

regexp = /^.+: (.+)$|^(.*)$/

'2021-03-31 12:12:05.856 LOG     : Message <checked [abc]>'.scan(regexp).first.compact
# => ["Message <checked [abc]>"]

'No Message was found'.scan(regexp).first.compact
# => ["No Message was found"]

Answer 2

I need to create a new field 'status' if the log field contains a specific string. I tried below code in fluentd but this doesnt work. I need to check if the log field contains the string 'error:' then the new field status should have error else if it has ok it should have ok.

<filter **>
  @type record_transformer
  enable_ruby true  
  <record>
   status ${record["log"].scan(/^.* ([[:<:]]error[[:>:]]|[[:<:]]ok[[:>:]]):.*$/i).first.compact} 
  </record>  
</filter>

The error I get is error = undefined method `compact' for nil:NilClass"