Combine Filebeat Logs to ship to Elastic Search based on unique Trace Id (Without Logstash)

Question

I have configured filebeat for shipping logs of my spring boot application directly to AWS Opensearch Service without configuration of logstash. I want to configure filebeat multiline regex in a way that it can combine all the logs in a single record having same transactionId. Can anyone please suggest ? Here the part of log configuration looks like in filebeat.yml:

filebeat.inputs:
- type: log
  paths:
    - E://filebeat//*.log
  reload.enabled: true
  reload.period: 300s
  enabled: true
  multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after
     
Log Sample:

2022-03-07 07:08:44 [ERROR] | transactionId=6af42925-48df-4f49-95a4-aa0e43b152ed | c.example.utility.advice.LoggingAdvice - Method Signature:  Boolean com.example.config.service.impl.ServiceImpl.testLogger() - Line No : 1345 - Exception :  null
2022-03-07 07:08:50 [ERROR] | transactionId=51a2d445-574b-4992-b867-a7fc5ee4b473 | c.example.utility.advice.LoggingAdvice - Method Signature:  Boolean com.example.config.service.impl.ServiceImpl.testLogger() - Line No : 1345 - Exception :  null
2022-03-07 05:26:08 [ERROR] | transactionId=17f4f815-ee74-4658-a8d7-347fea6dd9d1 | o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
    at com.example.config.service.impl.ServiceImpl.testLogger(EventServiceImpl.java:1345) ~[classes/:na]
    at com.example.config.service.impl.ServiceImpl$$FastClassBySpringCGLIB$$983a4fb2.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.9.jar:5.3.9]
    at 

Answer 1

You can use these lines in the multiline section in your filebeat.yml file,

multiline.type: pattern
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

For more reference please visit official_website .