氣流1.9.0 ExternalTask​​Sensor retry_delay = 30產生TypeError：無法腌制_thread.RLock對象

Question

如標題所述； 在Airflow 1.9.0中，如果您對ExternalTask​​Sensor使用retry_delay = 30（或任何其他數字）參數，則DAG運行得很好，直到您想在氣流GUI中清除任務實例->它將返回以下錯誤：“ TypeError：不能腌制_thread.RLock對象”（和一個不錯的消息），但是，如果您使用retry_delay = timedelta（seconds = 30），則清除任務實例效果很好。

如果我查看models.py方法，則Deepcopy應該很好，所以對我來說似乎是奇怪的行為。 我是否缺少某些東西，或者這是一個錯誤？

在下面，您可以找到一個最小的DAG示例。

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.sensors import ExternalTaskSensor
from datetime import datetime, timedelta


dag_name = 'soft_fail_example'
schedule_interval = "0 * * * *"
default_args = {
            'owner': 'airflow',
            'depends_on_past': False,
            'start_date': datetime(2018, 1, 1),
            'email': [],
            'email_on_failure': False,
            'email_on_retry': False
        }

test_dag = DAG(dag_name, default_args=default_args, schedule_interval=schedule_interval, 
catchup=False, max_active_runs=1)


ets = ExternalTaskSensor(task_id="test_external_task_sensor", dag=test_dag, soft_fail=False, 
timeout=10, retries=0, poke_interval=1, retry_delay=30, external_dag_id="dependent_dag_id",
                         external_task_id="dependent_task_id")

dummy_task = DummyOperator(task_id="collection_task", dag=test_dag)

dummy_task << ets

編輯：並根據要求的堆棧跟蹤：

    Ooops.

                              ____/ (  (    )   )  \___
                             /( (  (  )   _    ))  )   )\
                           ((     (   )(    )  )   (   )  )
                         ((/  ( _(   )   (   _) ) (  () )  )
                        ( (  ( (_)   ((    (   )  .((_ ) .  )_
                       ( (  )    (      (  )    )   ) . ) (   )
                      (  (   (  (   ) (  _  ( _) ).  ) . ) ) ( )
                      ( (  (   ) (  )   (  ))     ) _)(   )  )  )
                     ( (  ( \ ) (    (_  ( ) ( )  )   ) )  )) ( )
                      (  (   (  (   (_ ( ) ( _    )  ) (  )  )   )
                     ( (  ( (  (  )     (_  )  ) )  _)   ) _( ( )
                      ((  (   )(    (     _    )   _) _(_ (  (_ )
                       (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
                       ((__)        \\||lll|l||///          \_))
                                (   /(/ (  )  ) )\   )
                              (    ( ( ( | | ) ) )\   )
                               (   /(| / ( )) ) ) )) )
                             (     ( ((((_(|)_)))))     )
                              (      ||\(|(|)|/||     )
                            (        |(||(||)||||        )
                              (     //|/l|||)|\\ \     )
                            (/ / //  /|//||||\\  \ \  \ _)
    -------------------------------------------------------------------------------
    Node: jb-VirtualBox
    -------------------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/app.py", line 1988, in wsgi_app
        response = self.full_dispatch_request()
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/app.py", line 1641, in full_dispatch_request
        rv = self.handle_user_exception(e)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/app.py", line 1544, in handle_user_exception
        reraise(exc_type, exc_value, tb)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
        raise value
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/app.py", line 1639, in full_dispatch_request
        rv = self.dispatch_request()
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask/app.py", line 1625, in dispatch_request
        return self.view_functions[rule.endpoint](**req.view_args)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
        return self._run_view(f, *args, **kwargs)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
        return fn(self, *args, **kwargs)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/flask_login.py", line 755, in decorated_view
        return func(*args, **kwargs)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/www/utils.py", line 262, in wrapper
        return f(*args, **kwargs)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/www/utils.py", line 309, in wrapper
        return f(*args, **kwargs)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/www/views.py", line 989, in clear
        include_upstream=upstream)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/models.py", line 3527, in sub_dag
        dag = copy.deepcopy(self)
      File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
        y = copier(memo)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/models.py", line 3512, in __deepcopy__
        setattr(result, k, copy.deepcopy(v, memo))
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
        y[deepcopy(key, memo)] = deepcopy(value, memo)
      File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
        y = copier(memo)
      File "/home/jb/Documents/p3_cdc_data_flow/lib/python3.6/site-packages/airflow/models.py", line 2437, in __deepcopy__
        setattr(result, k, copy.deepcopy(v, memo))
      File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
        y = _reconstruct(x, memo, *rv)
      File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
        state = deepcopy(state, memo)
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
        y[deepcopy(key, memo)] = deepcopy(value, memo)
      File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
        y = _reconstruct(x, memo, *rv)
      File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
        state = deepcopy(state, memo)
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
        y[deepcopy(key, memo)] = deepcopy(value, memo)
      File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
        y = _reconstruct(x, memo, *rv)
      File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
        state = deepcopy(state, memo)
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
        y[deepcopy(key, memo)] = deepcopy(value, memo)
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
        append(deepcopy(a, memo))
      File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
        y = _reconstruct(x, memo, *rv)
      File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
        state = deepcopy(state, memo)
      File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
        y = copier(x, memo)
      File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
        y[deepcopy(key, memo)] = deepcopy(value, memo)
      File "/usr/lib/python3.6/copy.py", line 169, in deepcopy
        rv = reductor(4)
    TypeError: can't pickle _thread.RLock objects

Answer 1

再次查看此問題后，文檔明確指出retry_delay應該是一個timedelta。 因此，很幸運的是，如果您為retry_delay輸入一個整數而不是一個timedelta，那么DAG可以工作。

在models.py中，BaseOperator：

   :param retry_delay: delay between retries
   :type retry_delay: timedelta