[英]Use Logstash to get nested Airflow Logs, and send to Elasticsearch
I am new to Logstash and ELK as a whole.我是 Logstash 和 ELK 的新手。 I am trying to send my airflow logs to Logstash.
我正在尝试将我的 airflow 日志发送到 Logstash。 I am confused on how to configure my configuration file, especially because I have several (nested) log files.
我对如何配置我的配置文件感到困惑,特别是因为我有几个(嵌套的)日志文件。
My airflow is deployed on an AWS EC2 instance and my logs directory is something like this: /home/ubuntu/run/logs/scheduler/
我的 airflow 部署在 AWS EC2 实例上,我的日志目录是这样的:
/home/ubuntu/run/logs/scheduler/
The scheduler directory has a couple of dated folders.调度程序目录有几个过时的文件夹。 Using one of the folders as an example:
/home/ubuntu/run/logs/scheduler/2022-08-31.
以其中一个文件夹为例:
/home/ubuntu/run/logs/scheduler/2022-08-31.
The dated folder has files such as日期文件夹包含文件,例如
testing.py.log hello_world.py.log dag_file.py.log
Now, while configuring my /etc/logstash/conf.d/
(based on the log path I shared above), how can I define my path to pick all the logs?现在,在配置我的
/etc/logstash/conf.d/
(基于我上面共享的日志路径)时,如何定义我的路径来选择所有日志?
This is what my /etc/logstash/conf.d/apache-01.conf
currently looks like, but I know the path isn't accurate:这是我的
/etc/logstash/conf.d/apache-01.conf
目前的样子,但我知道路径不准确:
input {
file {
path => "~/home/ubuntu/run/log/scheduler/"
start_position => "beginning"
codec -> "line"
}
}
filter {
grok {
match => { "path" => "" }
}
mutate {
add_field => {
"log_id" => "%{[dag_id]}-%{[task_id]}-%{[execution_date]}-%{[try_number]}"
}
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
}
}
The path parameter needs a absolute path. path 参数需要一个绝对路径。 To process all py.log files you can use this input
要处理所有 py.log 文件,您可以使用此输入
input {
file {
path => "/home/ubuntu/run/logs/scheduler/*/*py.log"
start_position => "beginning"
codec -> "line"
}
}
To process only the files hello_world.py.log and dag_file.py.log you can use an array for your path要仅处理文件 hello_world.py.log 和 dag_file.py.log,您可以使用数组作为路径
input {
file {
path => ["/home/ubuntu/run/logs/scheduler/*/hello_world.py.log", "/home/ubuntu/run/logs/scheduler/*/dag_file.py.log"]
start_position => "beginning"
codec -> "line"
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.