[英]How to run a docker operator task from airflow which itself runs in a docker container?
我有一個 docker 容器在我的 windows 機器上運行,它是使用官方文檔中提供的 docker-compose 文件的改編版本構建的。
這工作正常,但我想移動 python 腳本,這些腳本是我的任務,從已安裝的plugins
文件夾中移出它們自己的 docker 容器。
為了對此進行測試,我創建了一個簡單的“Hello World:”示例腳本:
import numpy as np
def main():
print(f'Hello World')
print(np.random.random((3, 3)))
if __name__ == '__main__':
main()
連同以下簡單的 docker-file
FROM python:3.9
ADD main.py .
RUN pip install numpy
CMD ["python", "./main.py"]
我可以使用docker build -t docker-test-image.
使用docker run --name docker-test-container docker-test-image
從 CLI 運行它會給我預期的 output:
Hello World
[[0.20923763 0.25415024 0.95603957]
[0.01320074 0.58392589 0.24175036]
[0.06431375 0.87276564 0.9912474 ]]
到目前為止一切順利,但是如果我用我的 docker-operator 觸發 DAG,它就會失敗,我會得到幾個
FileNotFoundError: [Errno 2] 日志中沒有此類文件或目錄錯誤。
我的 DAG 腳本如下所示:
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import timedelta
with DAG(
dag_id= 'docker_test_dag',
description='Testing the docker operator',
schedule_interval=None,
start_date=days_ago(2),
catchup=False,
tags=['docker_test'],
default_args={
'owner': 'airflow',
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 0,
'depends_on_past': False,
'retry_delay': timedelta(minutes=5)
}
) as dag:
docker_test_task = DockerOperator(
task_id='docker_test_task',
image='docker-test-image',
api_version='auto',
auto_remove=True,
mount_tmp_dir=False,
container_name='docker-test-container',
command='echo "this is a test message shown from within the container',
docker_url='unix://var/run/docker.sock',
network_mode='bridge'
)
docker_test_task
仔細研究后,我認為這是一個 Docker-in-Docker 問題,最有可能的解決方案是在本教程中找到的。 我將- /var/run/docker.sock:/var/run/docker.sock
添加到我的 airflow docker-compose 文件中的卷部分。 不,DAG 仍然失敗,並出現以下日志錯誤:
追溯(最近一次通話最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py”,第 450 行,在發送超時=超時文件“/home/airflow/ .local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info() 2 File "/home/ airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py”,第 550 行,遞增 raise six.reraise(type(error), error, _stacktrace) File “/home/airflow/ .local/lib/python3.7/site-packages/urllib3/packages/six.py”,第 769 行,在 reraise raise value.with_traceback(tb) File “/home/airflow/.local/lib/python3.7/ site-packages/urllib3/connectionpool.py”,第 710 行,在 urlopen chunked=chunked,文件“/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py”,第 398 行,在 _make_request conn.request(method, url, **httplib_request_kw) 文件“/usr/local/lib/python3.7/http/client.py”, line 1281, in request self._send_request(method, 88 410139179088, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request self.endheaders(body, encode_chunked=encode_chunked) 文件 "/usr/local/ lib/python3.7/http/client.py", line 1276, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1036 ,在 _send_output self.send(msg) 文件“/usr/local/lib/python3.7/http/client.py”,第 976 行,在發送 self.connect() 文件“/home/airflow/.local/lib /python3.7/site-packages/docker/transport/unixconn.py", 第 30 行,在連接 sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission拒絕'))
在處理上述異常的過程中,又出現了一個異常:
追溯(最近一次調用最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 214 行,在 _retrieve_server_version return self.version(api_version=False )["ApiVersion"] 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py”,第 181 行,版本返回 self._result(self._get(url ), json=True) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py”,第 46 行,在內部返回 f(self, *args, ** kwargs) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 237 行,在 _get return self.get(url, **self._set_request_timeout(kwargs )) 文件“/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py”,第 542 行,在 get return self.request('GET', url, **kwargs) 文件中“/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py”,第 529 行,請求 resp = self.send(prep, **send_kwargs) 文件“/home/airflow/ .local/lib/python3.7/site-packages/requests/sessions.py",第 645 行,發送 r = adapter.send(request, **kwargs) 文件“/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py”,第 501 行,發送 raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('連接中止', PermissionError(13, 'Permission denied'))
在處理上述異常的過程中,又出現了一個異常:
追溯(最近一次通話最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py”,第 360 行,在執行 self.cli = self._get_cli() 文件“/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py”,第 390 行,在 _get_cli return APIClient(base_url=self. docker_url, version=self.api_version, tls=tls_config) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”, line 197, in init self._version = self._retrieve_server_version() 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 222 行,在 _retrieve_server_version f'Error while fetching server API version: {e }' docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
這應該怎么做?
終於找到答案了:
在 docker-compose 文件中,而不是
- /var/run/docker.sock:/var/run/docker.sock
在 windows 機器上使用它:
- //var/run/docker.sock:/var/run/docker.sock
小變化,大效果我猜..
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.