[英]Configuring Logstash filters
I have recently configured a ELK server, 我最近配置了ELK服务器,
In my app server (magento) var/log/ directory has many log files (including some 3rd party extension logs for magento), So I thought of sending all using *.log to logstash, because we are not aware of some of the log file names that might create in future because of 3rd party integration. 在我的应用程序服务器(magento)的var / log /目录中,有许多日志文件(包括magento的一些第三方扩展日志),所以我想到了将所有使用* .log发送到logstash的方法,因为我们不了解某些日志由于第三方集成而可能在将来创建的文件名。 and we need to catch those also.
我们也需要抓住那些。
Magento exception log has a multi-line logs with the stack trace, So What i did was add a filter using gork, Now It seems working (giving the concatenate output), Magento异常日志具有带堆栈跟踪的多行日志,所以我所做的是使用gork添加了一个过滤器,现在看来可以正常工作(给出连接输出),
Since I have one define type "staging-all-lincraft-logs" (in both config) all the log files are parsing through it (see the below code), 由于我有一个定义类型“ staging-all-lincraft-logs”(在两个配置中),所有日志文件都通过它进行解析(请参见以下代码),
I can't remove the *.log and give specific names sine un-aware of file names 我无法删除* .log并给特定名称正弦不知道文件名
Is there any way I can parse only the specific files(exception.log and system.log) in logstash config (I tried with adding the parth,It does not work) 有什么办法可以只解析logstash配置中的特定文件(exception.log和system.log)(我尝试添加parth,但不起作用)
logstash forwarder config: logstash转发器配置:
"files": [
{
"paths": [
"/home/deploy/lindcraft/current/codepool/var/log/*.log"
],
"fields": { "type": "staging-all-lincraft-logs" }
}
]
logstash filter config: logstash过滤器配置:
filter {
if [type] == "staging-all-lincraft-logs" {
multiline{
# path => "/home/deploy/lindcraft/current/codepool/var/log/exception.log"
pattern => "^%{TIMESTAMP_ISO8601:timestamp}"
what => "previous"
negate=> true
}
grok {
match => [
"message",
"(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:priority_name} \(%{INT:priority_level}\): %{GREEDYDATA:message}"
]
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
overwrite => [ "message" ]
}
date {
match => [ "timestamp", "ISO8601" ]
}
}
}
There is a workaround for you if you can change the log directory of 3rd party extensions. 如果可以更改第三方分机的日志目录,则有一个解决方法。 Then you can send all name unaware files to that new directory (eg. /var/log/new_directory/) and keep all other known logs in /var/log/ .
然后,您可以将所有不带名称的文件发送到该新目录(例如/ var / log / new_directory /) ,并将所有其他已知日志保留在/ var / log /中 。 So you change your logstash-forwader.conf file as below to send different types of log files to the logstash server.
因此,您可以如下更改logstash-forwader.conf文件,以将不同类型的日志文件发送到logstash服务器。
"files": [
{
"paths": [
"/home/deploy/lindcraft/current/codepool/var/log/syslog"
],
"fields": { "type": "syslog" }
},
{
"paths": [
"/home/deploy/lindcraft/current/codepool/var/log/exeption.log"
],
"fields": { "type": "exeption-log" }
},
{
"paths": [
"/home/deploy/lindcraft/current/codepool/var/log/new_directory/*.log"
],
"fields": { "type": "staging-all-lincraft-logs" }
}
]
And then add a filter to parse different log files with relevant GROK patterns. 然后添加一个筛选器以解析具有相关GROK模式的不同日志文件。 It will be some thing like below.
就像下面这样。
filter {
if [type] == "syslog" {
grok {
match => { "message" => "GROK pattern for syslog" }
}
}
else if [type] == "exeption-log" {
grok {
match => { "message" => "GROK pattern for exeption-log" }
}
}
else if [type] == "staging-all-lincraft-logs" {
grok {
match => { "message" => "GROK pattern for staging-all-lincraft-logs" }
}
}
}
I think if we need to treat log files differently in the logstash, we have to send them with specific types from logstash-forwarder. 我认为,如果需要在Logstash中对日志文件进行不同的处理,则必须从Logstash-Forwarder发送特定类型的日志文件。
Hope it helps.! 希望能帮助到你。!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.