I have been working on Apache Airflow for a while now to schedule my workflow. I seem to be having issues scheduling my DAG. I have been using this SO question for reference : Airflow not scheduling Correctly Python
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
from datetime import timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now() - timedelta(minutes=5),
'email': ['airflow@airflow.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
dag = DAG('dag_mkdir_folder', default_args=default_args,
schedule_interval=timedelta(minutes=5))
task_hello = BashOperator(task_id='print_hello',
bash_command='mkdir test_airflow', dag=dag)
I'm trying to run the task using the following list of commands :
airflow scheduler
airflow trigger_dag dag_mkdir_folder
I keep getting this as an error :
[2017-05-15 13:49:06,688] {models.py:322} DagFileProcessor406 INFO - Finding 'running' jobs without a recent heartbeat [2017-05-15 13:49:06,689] {models.py:328} DagFileProcessor406 INFO - Failing jobs without heartbeat after 2017-05-15 13:44:06.689284
The bash command is just supposed to create a new directory. The test version works fine.
Run the scheduler on a different terminal and then trigger your dag in another terminal
also try providing your full path where you want to make the directory. For example creating folder in the airflow directory:
task_hello = BashOperator(task_id='print_hello',
bash_command="mkdir ~/airflow/test_airflow", dag=dag)
This should create test_airflow folder inside airflow
Your current bash_command is telling Airflow to create a directory inside of the temporary directory that the DAG uses when it is running, which it blows away along with all of its contents after the DAG is run.
Consider changing directories to the directory you want to create it in.
A bash_command with that would look something like:
"cd <path_to_directory>; mkdir test_airflow"
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.