Regex Issue when log format has multiple ip's

Question

I have an issue with fluenTd log parser. The following config works fine when there are 2 ip's.

expression  /^(?<client_ip>[^ ]*)(?:, (?<lb_ip>[^ ]*))? (?<ident>[^ ]*) (?<user>[^ ]*) \[(?<time>[^ ]* [^ ]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) (?<protocol>[A-Z]{1,}[^ ]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)/

This matches:

148.165.41.129, 10.25.1.120 - - [09/Dec/2019:16:22:23 +0000] "GET /comet_request/44109669162/F1551019433002Y5MYEP?F155101943300742PMLG=1551019433877&_=1575904426457 HTTP/1.1" 200 0 0 0

When there are 3 ip's, i get a pattern not match warning.

This doesn't match :

176.30.235.70, 165.225.70.200, 10.25.1.120 - - [09/Dec/2019:13:30:57 +0000] \"GET /comet_request/71142769981/F1551018730440IY5YNF?F1551018721447ZVKYZ4=1551018733078&_=1575898029473 HTTP/1.1\" 200 0 0 0

I tried the following regex, but doesn't work.Can someone please help?

expression /^(?<client_ip>[^ ]*)(?:, (?<proxy_ip>[^ ]*))? (?:, (?<lb_ip>[^ ]*))? (?<ident>[^ ]*) (?<user>[^ ]*) \[(?<time>[^ ]* [^ ]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) (?<protocol>[A-Z]{1,}[^ ]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/

Answer 1

You need to match the IPs with a more specific pattern, like [\\d.]+ or [^, ]+ , and make sure you also match the last two fields (you are not matching them and $ requires the end of line/string).

Use a pattern like

^(?<client_ip>[^ ,]+)(?:, +(?<proxy_ip>[^ ,]+))?(?:, +(?<lb_ip>[^ ,]+))? (?<ident>[^ ]+) (?<user>[^ ]+) \[(?<time>[^\]\[ ]* [^\]\[ ]*)\] "(?<method>\S+)(?: +(?<path>\S+) (?<protocol>[A-Z][^" ]*)[^"]*)?" (?<code>\S+) (?<size>\S+) \S+ \S+$

See the regex demo

The IP matching part is ^(?<client_ip>[^ ,]+)(?:, +(?<proxy_ip>[^ ,]+))?(?:, +(?<lb_ip>[^ ,]+))? , see that [^ ,]+ matches 1+ chars other than a space and , and \\S+ \\S+ are added at the end of the pattern (if these are numbers, you may use \\d+ \\d+ and capture them if needed).

Answer 2

Example strings

Let's consider an abbreviated version of your question, focusing on the first four named ranges (as dealing with the remaining named ranges is straightforward).

str1 = "148.165.41.129, 10.25.1.120 - - [09/Dec/2019:16:22:23 +0000]"

str2 = "176.30.235.70, 165.225.70.200, 10.25.1.120 - - [09/Dec/2019:13:30:57 +0000]"

The regular expression written in free-spacing mode

The following regular expression can be used to extract the contents of the named ranges, provided the string has a valid structure. Notice that it requires IPv4 addresses and the date-time string to have valid patterns (rather than merely [^ ]+ and [^ ]+ [^ ]+ ). I've written the regular expression in free-spacing mode to make it self-documenting.

r = /
    \A              # match the beginning of the string 
    (?<client_ip>   # begin a capture group named client_ip
      \g<user_ip>   # evaluate the subexpression (capture group) named user_ip
    )               # end capture group client_ip
    (?:             # begin a non-capture group
      ,[ ]          # match the string ', '
      (?<lb_ip>     # begin a capture group named lb_ip
        \g<user_ip> # evaluate the subexpression (capture group) named user_ip
      )             # end capture group lb_ip
    )?              # end non-capture group and optionally execute it
    (?:             # begin a non-capture group
      ,[ ]          # match the string ', '
      (?<user_ip>   # begin a capture group named user_ip
        \d{1,3}     # match 1-3 digits 
        (?:         # begin a non-capture group
          \.\d{1,3} # match a period followed by 1-3 digits
        ){3}        # end the non-capture group and execute 3 times
      )             # end capture group user_id
    )               # end non-capture group
    [ ]-[ ]-[ ]\[   # match the string ' - - ['
    (?<time>        # begin a capture group named time 
      \d{2}\/\p{L}{3}\/\d{4}:\d{2}:\d{2}:\d{2}[ ]\+\d{4}
                    # match a time string
    )               # end capture group time                    
    \]              # match string ']'
    \z              # match end of string
    /x              # free-spacing regex definition mode

Match the strings against the regular expression

We now confirm the two strings match this regular expression and extract the contents of the capture groups.

    m1 = str1.match(r)
    m1.named_captures
      #=> {"client_ip"=>"148.165.41.129",
      #    "lb_ip"=>nil,
      #    "user_ip"=>"10.25.1.120",
      #    "time"=>"09/Dec/2019:16:22:23 +0000"} 

    m2 = str2.match(r)
    m2.named_captures
      #=> {"client_ip"=>"176.30.235.70",
      #    "lb_ip"=>"165.225.70.200",
      #    "user_ip"=>"10.25.1.120",
      #    "time"=>"09/Dec/2019:13:30:57 +0000"}

Subexpression Calls

Rather than replicating the content of the capture group user_ip for each of the first two named capture groups I have simply used \\g<user_ip> , which, in effect, tells the regex engine to evaluate the contents of capture group (subexpression) user_ip at the location where \\g<user_ip> is referenced. Search for "Subexpression Calls" in the docs for Regexp .

Notice that the subexpression calls are forward-looking . Suppose we instead wrote:

r = /
    \A 
    (?<client_ip>\d{1,3}(?:\.\d{1,3}){3})
    (?:,[ ](?<lb_ip>\g<client_ip>))?
    (?:,[ ](?<user_ip>\g<client_ip>))
    [ ]-[ ]-[ ]\[
    (?<time>\d{2}\/\p{L}{3}\/\d{4}:\d{2}:\d{2}:\d{2}[ ]\+\d{4}) 
    \]
    \z
    /x

Then

    m1 = str1.match(r)
    m1.named_captures
      #=> {"client_ip"=>"10.25.1.120",
      #    "lb_ip"=>nil,
      #    "user_ip"=>"10.25.1.120", 
      #    "time"=>"09/Dec/2019:16:22:23 +0000"}

    m2 = str2.match(r)
    m2.named_captures
      #=> {"client_ip"=>"10.25.1.120",
      #    "lb_ip"=>"165.225.70.200",
      #    "user_ip"=>"10.25.1.120",
      #    "time"=>"09/Dec/2019:13:30:57 +0000"} 

As seen, the contents of the capture group client_ip is set equal to the contents of user_ip . The reason for this behaviour is explained here (look for "In PCRE but not Perl, one interesting twist is..." and other referenced sections of that document).

The regular expression written conventionally

The regular expression is conventionally written as follows:

/\A(?<client_ip>\g<user_ip>)(?:, (?<lb_ip>\g<user_ip>))?(?:, (?<user_ip>\d{1,3}(?:\.\d{1,3}){3})) - - \[(?<time>\d{2}\/\p{L}{3}\/\d{4}:\d{2}:\d{2}:\d{2} \+\d{4})\]\z/

Notice that where there are spaces in the above there are character classes containing a single space when the regex is written in free-spacing mode. That is necessary because in free-spacing mode unprotected spaces are removed before the expression is parsed. Another way to protect spaces is to escape them ( \\ ). If it is desired to use whitespaces rather than spaces, \\s can be used.