[英]Parsing JSON event in Logstash
I have log in following format, it is a plain json with nested fields. 我以以下格式登录,它是带有嵌套字段的纯json。
{
"level": "info",
"message": {
"req": {
"headers": {
"host": "localhost:8080",
"connection": "keep-alive",
"x-forwarded-for": "192.168.1.1, 1.1.1.1",
"x-forwarded-proto": "http"
},
"url": "/products?userId=493d0aec-a9a7-42a3",
"method": "GET",
"originalUrl": "/products?userId=493d0aec-a9a7-42a3",
"params": {
"0": "/products"
},
"query": {
"userId": "493d0aec-a9a7-42a3"
},
"body": ""
},
"res": {
"headers": {
"traceid": "ac586e4e924048",
"x-correlation-id": "57d7920d-b623-48f8",
"content-type": "application/json;charset=UTF-8",
"content-length": "2",
"date": "Fri, 08 Mar 2019 09:55:45 GMT",
"connection": "close"
},
"statusCode": 200,
"body": "[]"
},
"gateway": "internal"
},
"correlationId": "57d7920d-b623-48f8",
"timestamp": "2019-03-08T09:55:45.833Z"
}
How can I parse it correctly using Filebeat and Logstash to see all json fields in Kibana as separate (parsed) fields? 如何使用Filebeat和Logstash正确解析它,以将Kibana中的所有json字段视为单独的(解析的)字段? I have a problem with "message" field which has nested json fields. 我有嵌套json字段的“ message”字段有问题。 I have no problem to parse an event which has string in "message", but not json. 我没有问题可以解析一个在“消息”中包含字符串但不是json的事件。
My attempts: 我的尝试:
1 . 1。 I tried to tell Filebeat that it is a json with following configuration: 我试图告诉Filebeat它是具有以下配置的json:
(and doing nothing on LS side) (并且在LS端不执行任何操作)
filebeat.inputs:
- type: stdin
json.keys_under_root: true
json.add_error_key: true
The result is strange for me, because I got "message" as a string in Kibana where all :
are replaced with =>
结果对我来说很奇怪,因为我在Kibana中以字符串形式获取了“消息”,其中所有:
均替换为=>
{
"req" => {
"originalUrl" => "/offers", "params" => {
"0" => "/offers"
}, "query" => {}, "body" => "", "headers" => {
"accept-encoding" => "gzip", "user-agent" => "okhttp/3.8.1", "x-consumer-id" => "f2a6e4cd-2224-4535
Other fields outside the "message" are parsed correctly 正确解析“消息”之外的其他字段
2 . 2。 I did nothing on Filebeat side and use filter in LS: 我没有在Filebeat方面执行任何操作,并在LS中使用了过滤器:
json {
source => "message"
target => "message_json"
}
Logs are not appeared in Kibana at all, I got following errors in LS: 日志根本没有出现在Kibana中,我在LS中遇到以下错误:
[2019-03-08T09:55:47,084][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. [2019-03-08T09:55:47,084] [WARN] [logstash.outputs.elasticsearch]无法将事件索引到Elasticsearch。 {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.5.0-2019.03.08-sdx", :_type=>"doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"filebeat-6.5.0-2019.03.08-sdx", "_type"=>"doc", " id"=>"ERS6XGkBgE -US7A6Mvt", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [json.message] of type [keyword]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:461"}}}}} [2019-03-08T09:55:47,085][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status => 400,:action => [“ index”,{:_id => nil,:_index =>“ filebeat-6.5.0-2019.03.08-sdx”,:_type =>“ doc”,: routing => nil},#],:response => {“ index” => {“ _ index” =>“ filebeat-6.5.0-2019.03.08-sdx”,“ _type” =>“ doc”,“ id “ =>” ERS6XGkBgE -US7A6Mvt“,”状态“ => 400,”错误“ => {”类型“ =>” mapper_parsing_exception“,”原因“ =>”无法解析类型为[关键字]的字段[json.message] ]“,” caused_by“ => {” type“ =>” illegal_state_exception“,”原因“ =>”无法在1:461的START_OBJECT上获取文本“”}}}} [2019-03-08T09:55 :47,085] [WARN] [logstash.outputs.elasticsearch]无法将事件编入Elasticsearch。 {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.5.0-2019.03.08-sdx", :_type=>"doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"filebeat-6.5.0-2019.03.08-sdx", "_type"=>"doc", " id"=>"EhS6XGkBgE -US7A6Mvt", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [json.message] of type [keyword]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:461"}}}}} {:status => 400,:action => [“ index”,{:_id => nil,:_index =>“ filebeat-6.5.0-2019.03.08-sdx”,:_type =>“ doc”,: routing => nil},#],:response => {“ index” => {“ _ index” =>“ filebeat-6.5.0-2019.03.08-sdx”,“ _type” =>“ doc”,“ id “ =>” EhS6XGkBgE -US7A6Mvt“,” status“ => 400,” error“ => {” type“ =>” mapper_parsing_exception“,”原因“ =>”无法解析类型为[keyword]的字段[json.message] ]“,” caused_by“ => {” type“ =>” illegal_state_exception“,”原因“ =>”无法在1:461的START_OBJECT上获取文字“”}}}}}
This filter works fine for me if the "message" field is a string (not a json). 如果“消息”字段是字符串(而不是json),则此过滤器对我来说效果很好。
Any ideas how to parse nested json in "message field"? 有什么想法如何解析“消息字段”中的嵌套json吗?
I have problems to parse json by logstash to. 我在通过logstash解析JSON时遇到问题。
I was struggling with this problem for a while. 我在这个问题上挣扎了一段时间。 And failed to solve into logstash. 并没有解决成logstash。
But fortunately, we have ingested node in elasticsearch itself. 但幸运的是,我们已经在弹性搜索本身中吸收了节点 。
I want to suggest my solution to your problem: 我想为您的问题提出我的解决方案:
You make pipeline (very simple pipeline): 您制作管道(非常简单的管道):
{
"description": "Parse JSON log",
"processors": [
{
"json": {
"field": "message",
"target_field": "message-json"
}
},
{
"remove": {
"field": "message"
}
},
{
"append": {
"field": "tags",
"value": [
"isjsonlog"
]
}
}
],
"on_failure": [
{
"append": {
"field": "tags",
"value": [
"nonjsonlog"
]
}
}
]
}
In your output plugin, you configure: 在输出插件中,配置:
elasticsearch {
hosts => [ localhost ]
index => "filebeat-%{+YYYY.MM.dd}"
manage_template => false
pipeline => "your_pipeline_name"
}
And you forget problems with json parsing. 而且您会忘记json解析的问题。
If you use filebeat you able to send json logs direct to pipeline by configuring filebeat: 如果使用filebeat ,则可以通过配置filebeat 将json日志直接发送到管道 :
output.elasticsearch:
....
pipelines:
- pipeline: "your pipeline name"
when:
contains:
message: "{"
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.