流利的日志解析缺少 java 堆栈跟踪日志的新行 \n

Question

i have the below fluentd config used to parse a java stacktrace log:

  @id fluentd-containers.log
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format multiline
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

And i was expecting receive that log like shown in the syslog, something like below:

java.lang.NullPointerException: Name is null
    at java.lang.Enum.valueOf(Enum.java:236)
    at sa.com.stcs.tracking.events.ProfileType.valueOf(ProfileType.java:3)
    at sa.com.stcs.geofence.GeofenceProfileRepository.mapToGeofenceProfile(GeofenceProfileRepository.java:110)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

but unfortunately it comes to kibana through elasticsearch with missing new line \n see pic.

Question: where exactly should i introduce the new line special char \n . ??

Answer 1

You should not use \n . If you look at the documentation here , format_firstline and format1 means fluentd will consider a new event (ie a new log message) when it finds something matching regex as format_firstline which should be date in your case. As soon as it finds the date, it parses the log message as per format1 . It wouldn't log a new line until a line with date is encountered so you don't have to worry about it.

• The following will parse time (ex: 2019-11-28 22:22:14 ), thread (ex: pool.thread.2 ), level (ex: INFO , ERROR etc) and message (ex. java.lang.... ) until it finds the next line with starting with date.

format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)

• If you want to keep it simple, you can only use following which will parse log time and everything else as message:

format_firstline /\d{4}-\d{1,2}-\d{1,2}/ format1 /^(?<logtime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{1,9}) (?<log>.*)/