[英]Logstash grok failing
我試圖找到一條消息,但它在日志中沒有_grokparsefailure,但實際上並沒有說明它失敗了。 grok查詢適用於https://grokdebug.herokuapp.com/
input {
file {
type => "apache-access"
path => "C:/prdLogs/sent/*"
}
filter {
grok {
match => ['message', '%{IP:clientip} - - \[%{GREEDYDATA:raw_timestamp} \] "%{WORD:httpmethod} %{NOTSPACE:referrer} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} "-" "%{NOTSPACE:request}" %{QS:UserAgent} %{WORD:httpmethodO} - - HTTP/%{NUMBER:httpversion2} "%{WORD:session}:%{WORD:httpmed}" "-" %{NUMBER:duration}' ]
}
date {
match => [ "raw_timestamp" , 'dd/MMM/yyyy:HH:mm:ss Z' ]
target => '@timestamp'
}
}
output {
elasticsearch { hosts => ["111.44.44.44:9200"] }
}
數據看起來像:
199.77.22.22 - - [26/Feb/2017:10:18:45 +0800] "GET /myapp/app/i18n/key/parent.selector.label.select.item/?locale=en_GB&dojo.preventCache=1488075524942 HTTP/1.1" 200 "-" "https://mywebsite.here.com:31000/myApp/home.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)" GET - - HTTP/1.1 "0000bKOk4n4SSBHuyJJKed085D6:1ap8u8p8j" "-" 3203
199.77.22.22 - - [26/Feb/2017:10:18:45 +0800] "GET /myapp/app/i18n/key/parent.selector.label.no.recently.used/?locale=en_GB&dojo.preventCache=1488075525483 HTTP/1.1" 200 "-" "https://mywebsite.here.com:31000/myApp/home.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)" GET - - HTTP/1.1 "0000bKOk4n4SSBHuyJJKed085D6:1ap8u8p8j" "-" 3159
199.77.22.22 - - [26/Feb/2017:10:18:46 +0800] "GET /myapp/app/i18n/key/selector.label.selected/?locale=en_GB&dojo.preventCache=1488075525843 HTTP/1.1" 200 "-" "https://mywebsite.here.com:31000/myApp/home.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)" GET - - HTTP/1.1 "0000bKOk4n4SSBHuyJJKed085D6:1ap8u8p8j" "-" 3600
199.77.22.22 - - [26/Feb/2017:10:18:46 +0800] "GET /myapp/app/i18n/key/actor.selector.label.remove.all/?locale=en_GB&dojo.preventCache=1488075526305 HTTP/1.1" 200 "-" "https://mywebsite.here.com:31000/myApp/home.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)" GET - - HTTP/1.1 "0000bKOk4n4SSBHuyJJKed085D6:1ap8u8p8j" "-" 3224
199.77.22.22 - - [26/Feb/2017:10:18:46 +0800] "GET /myapp/app/i18n/key/com.label.filter.objects/?locale=en_GB&dojo.preventCache=1488075526711 HTTP/1.1" 200 "-" "https://mywebsite.here.com:31000/myApp/home.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Tablet PC 2.0)" GET - - HTTP/1.1 "0000bKOk4n4SSBHuyJJKed085D6:1ap8u8p8j" "-" 3299
這實際上是一個apache訪問日志,但我無法使用COMBINEDAPACHELOG或COMMONAPACHELOG。 實際上是同樣的錯
elasticsearch中的所有條目都標記為“_grokparsefailure”。 我在調試模式下使用log.level運行logstash但是在日志中沒有看到任何錯誤。
我使用的是最新版本的logstash。
請指教。
R2 D2謝謝,我試過這個但沒有快樂:(
我創建了一個模式文件並粘貼了你的模式。 我剛剛將有效負載更改為“130.39.22.22 - [23 / Feb / 2015:10:18:45 +0800]”,以下是我的過濾器:
filter {
grok {
patterns_dir => ["c:/logstashconfig/patterns"]
match => ['message', '%{IP:clientip} - - /[%{DATE_CUSTOM:timestamp}/]']
}
date {
match => [ "timestamp" , 'dd/MMM/yyyy:HH:mm:ss Z' ]
target => '@timestamp'
}
}
logstash中的調試日志:
{
"path" => "C:/prdLogs/sent/test",
"@timestamp" => 2017-03-03T00:06:15.269Z,
"@version" => "1",
"host" => "hkw20012125",
"message" => "130.39.22.22 - - [23/Feb/2015:10:18:45 +0800]\r",
"type" => "apache-access",
"tags" => [
[0] "_grokparsefailure"
]
}
有任何想法嗎? 它是數據末尾的+0800嗎? 謝謝。
當你必須構建自己的模式時,從左側開始,慢慢地,並使用調試器 。
如果您測試此模式:
%{IP:clientip} - - \[
它有效,但這一個:
%{IP:clientip} - - \[%{GREEDYDATA:raw_timestamp} \]
沒有。 將模式與輸入進行比較表明時間戳和關閉括號之間沒有空格。
將此部分模式更改為:
%{IP:clientip} - - \[%{GREEDYDATA:raw_timestamp}\]
作品。
我想一旦你的模式中有GREEDYDATA
,就意味着從日志中考慮你的其余部分:
GREEDYDATA
的模式如下:
GREEDYDATA .* <-- means to capture the entire line
如果我沒有弄錯的話,你的grok匹配看起來應該是這樣的:
grok {
match => ['message', '%{IPV4:clientip} - - %{GREEDYDATA:data}']
}
除非您需要單獨提取值,否則上面的grok應該為您完成。 我認為你匹配timestamp
是錯誤的。 為了處理您的timestamp
您需要在模式文件中包含以下模式:
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
YEAR (?>\d\d){1,2}
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
DATE_CUSTOM %{MONTHDAY}[/]%{MONTH }[/]%{YEAR}:%{TIME}
然后你可以在你的grok
比賽中使用它:
grok {
match => ['message', '%{IPV4:clientip} - - \[%{DATE_CUSTOM:timestamp} %{GREEDYDATA:data}']
}
現在,您將能夠將timestamp
匹配為:
date {
match => [ "timestamp" , 'dd/MMM/yyyy:HH:mm:ss Z' ]
target => '@timestamp'
}
希望這可以幫助!
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.