[英]How can I parse plain text log following ECS (elastic common schema) with logstash?
I am using rsyslog to send plain text log to logstash.我正在使用 rsyslog 将纯文本日志发送到 logstash。 But I cannot assign data to host.name or host.ip fields by grok.
但是我无法通过 grok 将数据分配给 host.name 或 host.ip 字段。 The system through following error:
系统通过以下错误:
Could not index event to Elasticsearch.
无法将事件索引到 Elasticsearch。 {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2020.09.03", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x3273b8c], :response=>{"index"=>{"_index"=>"logstash-syslog-2020.09.03", "_type"=>"_doc", "_id"=>"i2hRU3QBeWqyaoMf1lgh", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Could not dynamically add mapping for field [host.ip]. Existing mapping for [host] must be of type object but found [text]."}}}}
{:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2020.09.03", :routing=>nil, :_type=>"_doc" }, #LogStash::Event:0x3273b8c], :response=>{"index"=>{"_index"=>"logstash-syslog-2020.09.03", "_type"=>"_doc", "_id"= >"i2hRU3QBeWqyaoMf1lgh", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"无法为字段 [host.ip] 动态添加映射。现有映射为 [ host] 必须是 object 类型,但找到了 [text]。"}}}}
I tried with [host][name] but had an error message:我尝试使用 [host][name] 但有一条错误消息:
Grok regexp threw exception {:exception=>"Could not set field 'name' on object '' to value ''.This is probably due to trying to set a field like [foo][bar] = someValuewhen [foo] is not either a map or a string" ...
Grok regexp 抛出异常 {:exception=>"Could not set field 'name' on object '' to value ''。这可能是由于尝试设置一个字段,如 [foo][bar] = someValuewhen [foo] is not地图或字符串”...
Here is grok configuration:这是 grok 配置:
grok {
match => { "message" => "<%{INT:log.syslog.priority}>%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:host.name} %{DATA:process.name}(?:\[%{POSINT:process.pid}\])?: %{GREEDYDATA:syslog_message}" }
}
My purpose is to parsing log messages according to ECS standard so that SIEM app can analyze these logs.我的目的是根据 ECS 标准解析日志消息,以便 SIEM 应用程序可以分析这些日志。
I think that rsyslog already sends along with the message
field another field called host
which is a simple string.我认为 rsyslog 已经与
message
字段一起发送了另一个名为host
字段,它是一个简单的字符串。 Then, when you try to assign something to [host][ip]
it fails, because [host]
is not an object but a string.然后,当您尝试将某些内容分配给
[host][ip]
它会失败,因为[host]
不是对象而是字符串。
What you can try to do is to rename the original host
field to something else and it should work:您可以尝试做的是将原始
host
字段重命名为其他名称,它应该可以工作:
Add this just before your grok
filter:在您的
grok
过滤器之前添加以下内容:
mutate {
rename => {
"[host]" => "[source][address]"
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.