如何抓取Airflow的错误日志并发邮件？

Question

We are trying to report the failures that occur during Airflow job execution and capture the exceptions from logs and send it in email.

Currently we display following things in the failure email written in pyt.

failure_msg = """
            :red_circle: Task Failed.
            *Dag*: {dag}  
            *Task*: {task}
            *Execution Time*: {exec_date}
            *Log Url*: {log_url} 
            """.format(
        dag=context.get('task_instance').dag_id,
        task=context.get('task_instance').task_id,
        ti=context.get('task_instance'),
        exec_date=context.get('execution_date'),
        log_url=context.get('task_instance').log_url

I was looking to capture the exception message from Airflow. The above message displays high level info like dag id, task id, url etc.

Referrer below Airflow documentation but so far did not get any way to capture exact exception message.

Currently I am manually throwing error in one of the DAG as

def scan_table():
    try:
        raise ValueError('File not parsed completely/correctly')
        logging.info(message)
    except Exception as error:
        raise ValueError('File not parsed completely/correctly inside exception block')
        print("Error while fetching data from backend", error)

Tried using this exception=context.get('task_instance').log.exception but it showed as

<bound method Logger.exception of <Logger airflow.task (INFO)>> 

In the DAG log output in Airflow UI, the exception is thrown as:

[2023-01-04, 09:05:07 UTC] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/bitnami/airflow/dags/git_airflow-dags/scan_table.py", line 37, in scan_table
    raise ValueError('File not parsed completely/correctly')
ValueError: File not parsed completely/correctly

I want to capture this part of log and print in the failure_msg in the Python snippet.

Answer 1

I was able to find answer to this.

We can get access to exception thrown by Airflow using below snipped.

context.get('exception')