聚合logstash過濾器配置

Question

我的目標是在logstash中基於pId組合事件。 但我發現具有相同pId的事件不會合並為一個事件。 添加聚合后，我無法看到任何更改。請幫忙

日志看起來像這樣：

June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request:
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12

這是我的配置：

filter {
grok{
match => { "message" => "%{DATESTAMP:log_timestamp} %{DATA:id} %{DATA:pId} %{DATA:ClassName} [%{LOGLEVEL:severity}] - %{GREEDYDATA:message}" }
}
if [message] =~ /Received request:/ {
aggregate {
task_id => "%{pId}"
code => "map['message'] = event['message']"
map_action => "create"
}
}
else if [message] =~ /Total time:^/ {
aggregate {
task_id => "%{pId}"
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']"
map_action => "update"
end_of_task => true
timeout => 120
}
}
else {
aggregate {
task_id => "%{pId}"
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']"
map_action => "update"
}
}
}

Answer 1

總結是那些過濾器，可真的很難得到正確的一個。 在很大程度上，因為Logstash是從螺栓設計為並行處理管道，所以過濾器堆棧中的每個aggregate調用對於管道是唯一的，並且您無法確定所有事件是否將通過相同的管道運行。 開箱即用，就是這樣。

如果使用-w 1參數運行logstash以強制所有內容通過單個管道，則會出現此行為。

在這種情況下，我建議改為在input上使用multiline編解碼器。 這會將所有日志整合在一個事件中，您可以稍后在過濾器階段進行分析。 當然，這假設這些多行事件中的每一個都同時被丟棄並且不會被多路復用。 如果你得到多路復用，那么聚合將需要失去你的並行性。

input {
  file {
    path => "/var/log/app/debug_logs.log"
    codec => multiline {
       pattern => "Received request:"
       negate => true
       what => previous
    }
  }
}

這將搜索與您的已Received request: 不匹配的事件Received request:正則表達式並將它們附加到上一行。 當它看到Received request:它將啟動一個新事件。 您的filter {}階段將會看到這一點

message => "June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request:\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12"

在並行上下文中操作更容易。