简体   繁体   English

基本 DAG 问题 - Bash 命令在终端中运行而不是 Airflow

[英]Basic DAG Issues - Bash Command Runs in Terminal and Not Airflow

I'm trying to run this DAG in Airflow with the intention of automating some python scripts.我试图在 Airflow 中运行这个 DAG,目的是自动化一些 python 脚本。 To test it I'm running a super simple.py as seen below.为了测试它,我正在运行一个超级简单的.py,如下所示。

When running 'python3 /home/j/git/J/Code_Modules/TEST.py' in my linux terminal in the Astro_Projects directory it works just fine.在 Astro_Projects 目录中的 linux 终端中运行“python3 /home/j/git/J/Code_Modules/TEST.py”时,它工作得很好。 Won't seem to run in the DAG though.不过似乎不会在 DAG 中运行。

Log file is returning "No such file or directory."日志文件返回“没有这样的文件或目录”。 The.py script is in my "Code_Modules" folder and the DAG is in my "Astro_Projects" folder as shown here: .py 脚本位于我的“Code_Modules”文件夹中,而 DAG 位于我的“Astro_Projects”文件夹中,如下所示:

在此处输入图像描述

I'm pretty new to this so any help is greatly appreciated.我对此很陌生,因此非常感谢任何帮助。 Thanks谢谢

#%%
import os
from datetime import datetime, timedelta
from airflow.models import DAG, Variable
from airflow.operators.bash_operator import BashOperator

# DAG Settings
args = {
    'owner': 'J',
    'start_date': datetime(2022, 9, 14),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
    'dagrun_timeout': timedelta(minutes=15),
    'email_on_failure': True,
    'email_on_retry': False,
    'catchup': False
}

dag = DAG(
    dag_id='TEST_DAG',
    default_args=args,
    schedule_interval='0 6 * * *',
    tags=['Prod','TEST']
)

send_reports = BashOperator(
    task_id='TEST',
    bash_command='python3 /home/j/git/J/Code_Modules/TEST.py',
    dag=dag
)

# %%

"TEST.py" is just this: “TEST.py”就是这样:

import os
print('thisworked')

use relative path:使用相对路径:

send_reports = BashOperator(
    task_id='TEST',
    bash_command='python3 ../Code_Modules/TEST.py',
    dag=dag
)

but it makes more sense to use PythonOperator to execute python functions,但使用 PythonOperator 执行 python 函数更有意义,

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM