Fluentbit Kubernetes - How to extract fields from existing logs

Question

I have configured EFK stack with Fluent-bit on my Kubernetes cluster. I can see the logs in Kibana.

I also have deployed nginx pod, I can see the logs of this nginx pod also in Kibana. But all the log data are sent to a single field "log" as shown below.

How can I extract each field into a separate field. There is a solution for fluentd already in this question. Kibana - How to extract fields from existing Kubernetes logs

But how can I achieve the same with fluent-bit?

I have tried the below by adding one more FILTER section under the default FILTER section for Kubernetes, but it didn't work.

 [FILTER] Name parser Match kube.* Key_name log Parser nginx

From this ( https://github.com/fluent/fluent-bit/issues/723 ), I can see there is no grok support for fluent-bit.

Answer 1

In our official documentation for Kubernetes filter we have an example about how to make your Pod suggest a parser for your data based in an annotation:

https://docs.fluentbit.io/manual/filter/kubernetes

Answer 2

Look at this configmap:

https://github.com/fluent/fluent-bit-kubernetes-logging/blob/master/output/elasticsearch/fluent-bit-configmap.yaml

The nginx parser should be there:

[PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z