气流任务以Bash方式工作，按计划失败

Question

I'm running:

Ubuntu 16.04
airflow v1.8.1
python 3.5

Airflow is running in a docker container.

I've got an airflow dag that is a single task -- a BashOperator that runs a python script:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from airflow.operators.sensors import ExternalTaskSensor
from airflow.operators import DummyOperator
from datetime import date, datetime, timedelta

start_date = date.today() - timedelta(1)

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(start_date.year, start_date.month, start_date.day),
    'retries': 0,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG('$MY_DAG_NAME', default_args=default_args, max_active_runs=1, schedule_interval="35 */2 * * *")
dag.catchup = False

t1 = BashOperator(dag=dag,
                  task_id='$TASK_1',
                  bash_command='python /airflow/scripts/$MY_PYTHON_SCRIPT.py')

t1

When I run python /airflow/scripts/$MY_PYTHON_SCRIPT.py at bash, it completes just fine. I monitor the memory usage with htop and I don't reach more than one third of total Mem -- the script maxes out at about 10% MEM% usage.

About two thirds of the time that I run this scheduled with airflow, however, I get the following error, seemingly randomly (one third of the time it will run no problem!):

[2018-08-22 07:36:33,979] {helpers.py:233} INFO - Terminating descendant processes of ['/opt/conda/envs/python35/bin/python', '/opt/conda/envs/python35/bin/airflow', 'run', '$MY_DAG_NAME', '$TASK_1', '2018-08-22T12:35:00', '--job_id', '650', '--raw', '-sd', 'DAGS_FOLDER/$MY_PYTHON_SCRIPT.py'] PID: 5200
[2018-08-22 07:36:33,979] {helpers.py:237} INFO - Terminating descendant process ['bash', '/tmp/airflowtmpwsq8ozwo/$TASK_122bzex5b'] PID: 5209
[2018-08-22 07:36:33,984] {helpers.py:195} ERROR - b''
[2018-08-22 07:36:33,984] {helpers.py:196} INFO - Killed process 5209 with signal 15
[2018-08-22 07:36:33,984] {helpers.py:237} INFO - Terminating descendant process ['python', '/airflow/scripts/$MY_PYTHON_SCRIPT.py'] PID: 5210
[2018-08-22 07:36:33,989] {helpers.py:195} ERROR - b''
[2018-08-22 07:36:33,989] {helpers.py:196} INFO - Killed process 5210 with signal 15
[2018-08-22 07:36:33,989] {helpers.py:242} INFO - Waiting up to 60s for processes to exit...
[2018-08-22 07:36:34,327] {base_task_runner.py:98} INFO - Subtask: [2018-08-22 07:36:34,326] {bash_operator.py:105} INFO - Command exited with return code -15
[2018-08-22 07:36:34,335] {models.py:1595} ERROR - Bash command failed
Traceback (most recent call last):
  File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/models.py", line 1493, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/operators/bash_operator.py", line 109, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2018-08-22 07:36:34,336] {models.py:1624} INFO - Marking task as FAILED.
[2018-08-22 07:36:34,352] {models.py:1644} ERROR - Bash command failed
[2018-08-22 07:36:34,353] {base_task_runner.py:98} INFO - Subtask: /opt/conda/envs/python35/lib/python3.5/site-packages/airflow/utils/helpers.py:351: DeprecationWarning: Importing DummyOperator directly from <module 'airflow.operators' from '/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/operators/__init__.py'> has been deprecated. Please import from '<module 'airflow.operators' from '/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/operators/__init__.py'>.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2018-08-22 07:36:34,353] {base_task_runner.py:98} INFO - Subtask:   DeprecationWarning)
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask: Traceback (most recent call last):
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask:   File "/opt/conda/envs/python35/bin/airflow", line 27, in <module>
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask:     args.func(args)
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask:   File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/bin/cli.py", line 392, in run
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask:     pool=args.pool,
[2018-08-22 07:36:34,354] {base_task_runner.py:98} INFO - Subtask:   File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask:     result = func(*args, **kwargs)
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask:   File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/models.py", line 1493, in _run_raw_task
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask:     result = task_copy.execute(context=context)
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask:   File "/opt/conda/envs/python35/lib/python3.5/site-packages/airflow/operators/bash_operator.py", line 109, in execute
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask:     raise AirflowException("Bash command failed")
[2018-08-22 07:36:34,355] {base_task_runner.py:98} INFO - Subtask: airflow.exceptions.AirflowException: Bash command failed
[2018-08-22 07:36:34,363] {helpers.py:245} INFO - Done waiting

Anyone come across this before or have any debugging tips? It's driving me nuts.

Thanks!

Answer 1

Your python script is getting run - but it has a non-zero return code so that means it's getting some kind of exception in that script.

The Error b"" suggests some kind of encoding problem with bytes.

I'd say the best way to debug this is put some kind of logging in your python code - logging.info("at point a in code").

That said - you might get better debugging and stack traces if you use a PythonOperator and put your python code in there.