[英]logstash grok multiline - how to merge to previous line any line that doesn't start with timestamp
sometimes I print to log indented pretty jsons which printed in multiple lines. 有时我会打印记录多行缩进的漂亮json。 so I need to be able to tell logstash to append these prints to the original line of the original event.
因此,我需要能够告诉logstash将这些打印内容附加到原始事件的原始行中。
example: 例:
xxx p:INFO d:2015-07-21 11:11:58,906 sourceThread:3iMind-Atlas-akka.actor.default-dispatcher-2 queryUserId: queryId: hrvJobId:6c1a4d60-e5e6-40d8-80aa-a4dc00e9f0c4 etlStreamId:70 etlOmdId: etlDocId: logger:tim.atlas.module.etl.mq.MQConnectorEtl msg:(st:Consuming) received NotifyMQ. sending to [openmind_exchange/job_ack] message:
{
"JobId" : "6c1a4d60-e5e6-40d8-80aa-a4dc00e9f0c4",
"Time" : "2015-07-21T11:11:58.904Z",
"Errors" : [ ],
"FeedItemSchemaCounts" : {
"Document" : 1,
"DocumentMetadata" : 1
},
"OtherSchemaCounts" : { }
}
Since I've set a special log4j appender to function solely as logstash input, this task should be quiet easy. 由于我已经设置了一个特殊的log4j附加程序,使其仅可以用作logstash输入,因此此任务应该很简单。 I control the layout of the log, so I can add as many prefix/suffix indicators as I please.
我控制日志的布局,因此我可以根据需要添加尽可能多的前缀/后缀指示符。
here's how my appender look like: 这是我的附加程序的外观:
log4j.appender.logstash-input.layout.ConversionPattern=xxx p:%p d:%d{yyyy-MM-dd HH:mm:ss,SSS}{UTC} sourceThread:%X{sourceThread} queryUserId:%X{userId} queryId:%X{queryId} hrvJobId:%X{hrvJobId} etlStreamId:%X{etlStreamId} etlOmdId:%X{etlOmdId} etlDocId:%X{etlDocId} logger:%c msg:%m%n
as you can see I've prefixed every message with 'xxx' so I could tell logstash to append any line which doesn't start with 'xxx' to the previous line 如您所见,我在每个消息前面都加上了“ xxx”,因此我可以告诉logstash在前一行后面加上任何不以“ xxx”开头的行
here's my logstash configuration: 这是我的logstash配置:
if [type] == "om-svc-atlas" {
grok {
match => [ "message" , "(?m)p:%{LOGLEVEL:loglevel} d:%{TIMESTAMP_ISO8601:logdate} sourceThread:%{GREEDYDATA:sourceThread} queryUserId:%{GREEDYDATA:userId} queryId:%{GREEDYDATA:queryId} hrvJobId:%{GREEDYDATA:hrvJobId} etlStreamId:%{GREEDYDATA:etlStreamId} etlOmdId:%{GREEDYDATA:etlOmdId} etlDocId:%{GREEDYDATA:etlDocId} logger:%{GREEDYDATA:logger} msg:%{GREEDYDATA:msg}" ]
add_tag => "om-svc-atlas"
}
date {
match => [ "logdate" , "YYYY-MM-dd HH:mm:ss,SSS" ]
timezone => "UTC"
}
multiline {
pattern => "<please tell me what to put here to tell logstash to append any line which doesnt start with xxx to the previous line>"
what => "previous"
}
}
yes it was easy indeed : 是的,确实很容易:
if [type] == "om-svc-atlas" {
grok {
match => [ "message" , "(?m)p:%{LOGLEVEL:loglevel} d:%{TIMESTAMP_ISO8601:logdate} sourceThread:%{GREEDYDATA:sourceThread} queryUserId:%{GREEDYDATA:userId} queryId:%{GREEDYDATA:queryId} hrvJobId:%{GREEDYDATA:hrvJobId} etlStreamId:%{GREEDYDATA:etlStreamId} etlOmdId:%{GREEDYDATA:etlOmdId} etlDocId:%{GREEDYDATA:etlDocId} logger:%{GREEDYDATA:logger} msg:%{GREEDYDATA:msg}" ]
add_tag => "om-svc-atlas"
}
date {
match => [ "logdate" , "YYYY-MM-dd HH:mm:ss,SSS" ]
timezone => "UTC"
}
multiline {
pattern => "^(?!xxx).+"
what => "previous"
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.