终止从 Airflow 到 spark/yarn 的进程信号

Question

Do someone know how next feature can be implemented? When you marked task in airflow as failed this stops Airflow process however this didn't stoped application running in yarn.

Something similar to

except KeyboardInterrupt:

when the application running in terminal.

Answer 1

You can call functions either on success or on failure by using Callback concepts https://airflow.apache.org/docs/apache-airflow/2.2.2/logging-monitoring/callbacks.html