[英]Can't set airflow's celery_result_backend setting to 'rpc://'
[英]Airflow Scheduler Crashes when setting Postgres celery result_backend
我嘗試使用 CeleryExecutor 實現 Apache Airflow。 對於數據庫,我使用 Postgres,對於 celery 消息隊列,我使用 Redis。 使用 LocalExecutor 時一切正常,但是當我在 airflow.cfg 中設置 CeleryExecutor 並希望將 Postgres 數據庫設置為 result_backend 時
result_backend = postgresql+psycopg2://airflow_user:*******@localhost/airflow
無論觸發哪個 DAG,在運行 Airflow 調度程序時都會收到此錯誤:
[2020-03-18 14:14:13,341] {scheduler_job.py:1382} ERROR - Exception when executing execute_helper
Traceback (most recent call last):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'backend'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _execute
self._execute_helper()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1441, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1503, in _validate_and_run_task_instances
self.executor.heartbeat()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/executors/base_executor.py", line 130, in heartbeat
self.trigger_tasks(open_slots)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 205, in trigger_tasks
cached_celery_backend = tasks[0].backend
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/local.py", line 146, in __getattr__
return getattr(self._get_current_object(), name)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/task.py", line 1037, in backend
return self.app.backend
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/base.py", line 1227, in backend
return self._get_backend()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/base.py", line 944, in _get_backend
self.loader)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/backends.py", line 74, in by_url
return by_name(backend, loader), url
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/backends.py", line 60, in by_name
backend, 'is a Python module, not a backend class.'))
celery.exceptions.ImproperlyConfigured: Unknown result backend: 'postgresql'. Did you spell that correctly? ('is a Python module, not a backend class.')
指向數據庫的完全相同的參數有效
sql_alchemy_conn = postgresql+psycopg2://airflow_user:*******@localhost/airflow
將 Redis 設置為 celery result_backend 有效,但我讀到它不是推薦的方式。
result_backend = redis://localhost:6379/0
有沒有人看到我做錯了什么?
您需要在數據庫連接字符串中添加db+
前綴:
f"db+postgresql+psycopg2://{user}:{password}@{host}/{database}"
文檔中也提到了這一點: https : //docs.celeryproject.org/en/stable/userguide/configuration.html#database-url-examples
您需要在數據庫連接字符串中添加db+
前綴:
result_backend = db+postgresql://airflow_user:*******@localhost/airflow
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.