[英]create a Grok filter for a specific date pattern in logstash
I am trying to use ELK stack to store old logs. 我正在尝试使用ELK堆栈来存储旧日志。 ** This is not a duplicate question.
**这不是重复的问题。 Please read below for details.
请阅读下面的详细信息。 ** I want to parse timestamp from my message which looks like below:
**我想从邮件中解析时间戳,如下所示:
Apr 1 04:01:04 i-b73lj53l journal: 152.17.62.1 - - [31/Mar/2017:20:01:04 +0000] "GET /api/people/5913b19b31b0f601004875a5?access_token=rNL7S4A2o5BdbX1QDxbL9L5Vx7j60kGIIhQ1tk9yDYRjUf5e8OKzGGnIDTrMXr5n&filter=%7B%22order%22%3A%22createdAt%20DESC%22%2C%22include%22%3A%5B%7B%22relation%22%3A%22friendships%22%2C%22scope%22%3A%7B%22where%22%3A%7B%22trashedAt%22%3A%7B%22exists%22%3Afalse%7D%7D%2C%22include%22%3A%5B%22 HTTP/1.1" 200 346 " http://api.mywebsite.com/ " "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4549.400 Mozilla/9.7.12900.400"
4月1日04:01:04 i-b73lj53l日记:152.17.62.1--[31 / Mar / 2017:20:01:04 +0000]“ GET / api / people / 5913b19b31b0f601004875a5?access_token = rNL7S4A2o5BdbX1QDxbL9L5Vx7j60kGyy %% R9nKnf%R9nKnf%R2R9n5%R2R9n5%B2B0E0%B2B0E0%B1B0F0%B2B0B0B0B0B0B0B0B0B0B0 22%3A%22createdAt%20DESC%22%2C%22include%22%3A%5B%7B%22relation%22%3A%22friendship%22%2C%22scope%22%3A%7B%22where%22%3A%7B% 22trashedAt%22%3A%7B%22exists%22%3Afalse%7D%7D%2C%22include%22%3A%5B%22 HTTP / 1.1“ 200 346” http://api.mywebsite.com/ “” Mozilla / 5.0(Windows NT 6.3; WOW64)AppleWebKit / 537.36(KHTML,例如Gecko)Chrome / 53.0.2785.104 Safari / 537.36 Core / 1.53.4549.400 Mozilla / 9.7.12900.400“
I have tried already for more than 20 hours and it seems my configuration is not being read at all because even if I add add_field or remove_field , there is no change in data. 我已经尝试了20多个小时 ,并且似乎根本没有读取我的配置,因为即使我添加add_field或remove_field ,数据也没有变化。
I have already enabled system logs as per the documentation filebeat documentation.
我已经按照文档filebeat文档启用了系统日志。
My std output looks like this: 我的std输出看起来像这样:
DEBUG [publish] pipeline/processor.go:275 Publish event:
{
"@timestamp": "2018-04-05T18:53:08.817Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.3"
},
"source": "/Users/garry/project/sampel",
"offset": 3231104,
"tags": [
"message-log"
],
"prospector": {
"type": "log"
},
"fields": {
"env": "dev"
},
"beat": {
"name": "Garry-MacBook-Pro-2.local",
"hostname": "Garry-MacBook-Pro-2.local",
"version": "6.2.3"
},
"message": "Apr 1 04:01:04 i-b73lj53l journal: 152.17.62.1 - - [31/Mar/2017:20:01:04 +0000] "GET /api/people/5913b19b31b0f601004875a5?access_token=rNL7S4A2o5BdbX1QDxbL9L5Vx7j60kGIIhQ1tk9yDYRjUf5e8OKzGGnIDTrMXr5n&filter=%7B%22order%22%3A%22createdAt%20DESC%22%2C%22include%22%3A%5B%7B%22relation%22%3A%22friendships%22%2C%22scope%22%3A%7B%22where%22%3A%7B%22trashedAt%22%3A%7B%22exists%22%3Afalse%7D%7D%2C%22include%22%3A%5B%22 HTTP/1.1" 200 346 "http://api.mywebsite.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4549.400 Mozilla/9.7.12900.400"\"
}
My current config is: 我当前的配置是:
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z", "d/MMM/yyyy:HH:mm:ss Z" ]
}
}
Not sure what your grok filter is doing, but your log is a syslog
so you can simply create a filter using %{SYSLOGLINE}
不知道您的grok过滤器在做什么,但是您的日志是
syslog
因此您可以使用%{SYSLOGLINE}
you can then parse, [31/Mar/2017:20:01:04 +0000]
which is stored in a message field
as follows, 然后,您可以解析
[31/Mar/2017:20:01:04 +0000]
,该消息存储在消息field
,如下所示,
\[%{MONTHDAY:monthday}/%{MONTH:month}/%{YEAR:year}:%{TIME:time}.%{ISO8601_TIMEZONE:Timezone}\]
which will produce following output, 将产生以下输出,
{
"monthday": [
"31"
],
"month": [
"Mar"
],
"year": [
"2017"
],
"time": [
"20:01:04"
],
"HOUR": [
"20",
"00"
],
"MINUTE": [
"01",
"00"
],
"SECOND": [
"04"
],
"Timezone": [
"+0000"
]
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.