如何使用 Airflow 在 python 中运行 bash 脚本

Question

I have a python function

def running_dump():
    with open('dags/scripts/shell_scripts/daily_pg_dump.sh', 'rb') as file:
        script = file.read()
        print(script)
    subprocess.call(script, shell=True)

and a shell file daily_pg_dump.sh

PGPASSWORD='*******' pg_dump -h ***** -p ***** -U ***** -d * -t table_1 > dags/data_bucket/table_1_backup.sql

Airflow Dag

pg_dump_to_storage = PythonOperator(
        task_id='task_1',
        python_callable=running_dump,
        dag=dag
    )

When I call the python function using Airflow, the shell script seems not to run because table_1_backup.sql is not created. Instead I'm getting Returned value was: 0 but no error appear. What I'm I missing?

Answer 1

If you want to run bash scripts from Airflow, you can use BashOperator instead of PythonOperator .

From this example in the documentation, in your case it would be:

from airflow.operators.bash import BashOperator

running_dump = “path/to/daily_pg_dump.sh ” # note the space after the script's name

pg_dump_to_storage = BashOperator(
   task_id='task_1', 
   bash_command=running_dump,
   dag=dag,
)

NOTE: You only need the space after the script's path name when you are NOT using templating in your bash script.