[英]Grok and mutate in Logstash filter gives no effect to the output
Logstash 將通過遠程主機上的 Filebeat 收集信息。 我最終的預期結果是創建一個圖表來顯示每天從 HTTP 日志的響應時間。 所以我關注日志消息中的響應時間。
現在從 Kibana 得到的 json 消息看起來像這樣
{
"_index": "filebeat-7.9.2-2020.10.06-000001",
"_type": "_doc",
"_id": "NYaZB3UB-JR2hYtrwawX",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2020-10-08T09:45:17.358Z",
"agent": {
"id": "acbc8cf5-7bc1-46ab-a9ad-8ee1fef26183",
"name": "42dcb5cf4d4a",
"type": "filebeat",
"version": "7.9.2",
"hostname": "42dcb5cf4d4a",
"ephemeral_id": "e45f89ee-f355-4490-b79b-394a2226ffe7"
},
"log": {
"offset": 5157015,
"file": {
"path": "/usr/logs/tomcat/test.com.access.2020-10-08.log"
}
},
"message": "203.149.37.226 - - [08/Oct/2020:11:45:10 +0200] \"POST /TestserviceWarning/get_warning/render?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482 HTTP/1.1\" 200 1151 0.010",
"tags": [
"tomcat"
],
"input": {
"type": "log"
},
"ecs": {
"version": "1.5.0"
},
"host": {
"name": "42dcb5cf4d4a"
}
},
"fields": {
"@timestamp": [
"2020-10-08T09:45:17.358Z"
]
},
"sort": [
1602150317358
]
}
和我的管道配置
input {
beats {
port => 5000
}
}
filter {
if "tomcat" in [tags] {
grok {
match => { "message" => "%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}
overwrite => [ "message" ]
}
mutate { add_field => {"respones_time" => "%{response_time_sec}"}
}
}
}
管道觸發后,我在 json 消息中沒有看到任何其他字段。 結果保持不變。
編輯於 2020 年 9 月 10 日
浪費了一整天的時間,我仍然無法讓它工作。
所以我打開了 DEBUG 日志級別,看起來 Logstash 能夠訪問過濾器,但它以某種方式沒有將過濾器應用於 output。
org.logstash.config.ir.compiler.ComputeStepSyntaxElement@8d7da797
P[filter-grok{"match"=>{"message"=>"%{IPORHOST:client} %{DATA} %{DATA:user} \\[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\\] \\\"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\\\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}}|[file]/usr/share/logstash/pipeline/logstash.conf:9:17:```
grok {
match => { "message" => "%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}
}
```]
into
org.logstash.config.ir.compiler.ComputeStepSyntaxElement@8d7da797
然后我看到了新的異常信息
[INFO ] 2020-10-09 06:24:55.568 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-10-09 06:24:55.649 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
和
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @72da5f0e
您的 logstash 配置看起來正確。 下面是來自 logstash 的 output,同樣應該在 Elastic 搜索中獲取。 我假設您在輸入最新配置后重新攝取了日志文件。 此外,確保重新加載 logstash 配置並刷新 Kibana 中的索引,以便識別最新字段。
{
"response_time_sec" => "0.010",
"timezone" => "+0200",
"@timestamp" => 2020-10-08T13:10:21.115Z,
"host" => "4ddee8887e1b",
"client" => "203.149.37.226",
"respones_time" => "0.010",
"@version" => "1",
"bytes" => "1151",
"path" => "/usr/share/logstash/stack/data/data.log",
"logtimestamp" => "08/Oct/2020:11:45:10",
"method" => "POST",
"uri_path" => "/TestserviceWarning/get_warning/render",
"user" => "-",
"protocol" => "HTTP/1.1",
"code" => "200",
"params" => "?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482",
"message" => "203.149.37.226 - - [08/Oct/2020:11:45:10 +0200] \"POST /TestserviceWarning/get_warning/render?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482 HTTP/1.1\" 200 1151 0.010"
}
盡管您可以通過以下方式簡單地完成您想要實現的目標
filter {
grok{
match => { "message" => "%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time}"}
}
}
output 字段想
{
"bytes" => "1151",
"@version" => "1",
"logtimestamp" => "08/Oct/2020:11:45:10",
"method" => "POST",
"code" => "200",
"client" => "203.149.37.226",
"host" => "169f8e2acdce",
"timezone" => "+0200",
"params" => "?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482",
"path" => "/usr/share/logstash/stack/data/data.log",
"@timestamp" => 2020-10-08T13:15:31.576Z,
"uri_path" => "/TestserviceWarning/get_warning/render",
"user" => "-",
"protocol" => "HTTP/1.1",
"response_time_sec" => "0.010",
"message" => "203.149.37.226 - - [08/Oct/2020:11:45:10 +0200] \"POST /TestserviceWarning/get_warning/render?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482 HTTP/1.1\" 200 1151 0.010"
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.