Use Logstash to get nested Airflow Logs, and send to Elasticsearch

Question

I am new to Logstash and ELK as a whole. I am trying to send my airflow logs to Logstash. I am confused on how to configure my configuration file, especially because I have several (nested) log files.

My airflow is deployed on an AWS EC2 instance and my logs directory is something like this: /home/ubuntu/run/logs/scheduler/

The scheduler directory has a couple of dated folders. Using one of the folders as an example: /home/ubuntu/run/logs/scheduler/2022-08-31.

The dated folder has files such as

testing.py.log hello_world.py.log dag_file.py.log

Now, while configuring my /etc/logstash/conf.d/ (based on the log path I shared above), how can I define my path to pick all the logs?

This is what my /etc/logstash/conf.d/apache-01.conf currently looks like, but I know the path isn't accurate:

input {
        file {
                path => "~/home/ubuntu/run/log/scheduler/"
                start_position => "beginning"
                codec -> "line"
        }
}

filter {
  grok {
    match => { "path" => "" }
  }
  mutate {
      add_field => {
        "log_id" => "%{[dag_id]}-%{[task_id]}-%{[execution_date]}-%{[try_number]}"
      }
  }
}
output{
        elasticsearch {
                hosts => ["localhost:9200"]
        }
}

Answer 1

The path parameter needs a absolute path. To process all py.log files you can use this input

input {
    file {
            path => "/home/ubuntu/run/logs/scheduler/*/*py.log"
            start_position => "beginning"
            codec -> "line"
    }
}

To process only the files hello_world.py.log and dag_file.py.log you can use an array for your path

    input {
    file {
            path => ["/home/ubuntu/run/logs/scheduler/*/hello_world.py.log", "/home/ubuntu/run/logs/scheduler/*/dag_file.py.log"]
            start_position => "beginning"
            codec -> "line"
    }
}