fluentbit reporting each line of the java stacktrace log as new log event

Question

any help is greatly appreciated.

I have fluentd running on kubernetes. I have logs coming from java application. The stack trace is multiline and the issue is each line of the multiline log is reported as a new log event.

below is my config. can someone advice what is wrong with the config? additionally, I am using containerd and not docker for running containers.

input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Multiline         On
        Parser_Firstline  multiline
        Parser_1          cri

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://k3s-master.test.com:6443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
    [FILTER]
        Name      parser
        Match     kube.*
        Key_Name  log
        Parser    cri
        Reserve_Data On
    [FILTER]
        Name      parser
        Match     kube.*
        Key_Name  message
        Parser    json
        Reserve_Data On
        Preserve_Key On

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Logstash_Prefix logstash-k3s-test
        Replace_Dots    On
        Retry_Limit     False
        Type            flb_type



parsers.conf: |
    [PARSER]
        Name   json
        Format json

[PARSER]
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

[PARSER]
    Name        multiline
    Format      regex
    Regex       ^(?<time>\d{4}[-\/]\d{2}[-\/]\d{2}.*) (?<stream>stdout|stderr) . (?<message>(\{|(ts=)?\[?\d{4}[-\/]\d{2}[-\/]\d{2}).*)$

sample log:

2020-10-15T08:48:37.932000736+02:00 stdout F 2020-10-15 08:48:37.923 ERROR [Service,a1793c4339d3f230,a1793c4339d3f230,true] 1 --- [nio-8080-exec-2] c.k.m.i.rest.GlobalExceptionHandler      : Status Response Error: 304 NOT_MODIFIED
2020-10-15T08:48:37.932050633+02:00 stdout F
2020-10-15T08:48:37.932061926+02:00 stdout F org.springframework.web.server.ResponseStatusException: 304 NOT_MODIFIED
2020-10-15T08:48:37.932071245+02:00 stdout F    at com.kobo.microservice.rest.InventoryController.queryInventory(InventoryController.java:70) ~[classes!/:1.197.1]
2020-10-15T08:48:37.932080389+02:00 stdout F    at jdk.internal.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932088907+02:00 stdout F    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932096876+02:00 stdout F    at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932113131+02:00 stdout F    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932121817+02:00 stdout F    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932131116+02:00 stdout F    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932140307+02:00 stdout F    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]

Answer 1

According to fluentd documentation :

Additional Step 2: Concatenate Multiple Lines Log Messages

The application log is stored in the log field of the record. You can concatenate these logs by using fluent-plugin-concat filter before sending it to the destinations.

 <filter docker.**> @type concat key log stream_identity_key container_id multiline_start_regexp /^-e:2:in `\\/'/ multiline_end_regexp /^-e:4:in/ </filter>

Original Events:

 2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'"} 2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:2:in `do_division_by_zero'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"} 2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:4:in `<main>'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}

Filtered Events:

 2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'\\ne:2:in `do_division_by_zero'\\ne:4:in `<main>'"}

If the logs are typical stacktraces, consider using detect-exceptions plugin instead.