[英]How to run a docker operator task from airflow which itself runs in a docker container?
I have a docker container running on my windows machine, which was build with an adapted version of the docker-compose file provided in the official docs .我有一个 docker 容器在我的 windows 机器上运行,它是使用官方文档中提供的 docker-compose 文件的改编版本构建的。
This works fine, however I would like to move the python scripts, which are my tasks out of the mounted plugins
folder and into their own docker containers.这工作正常,但我想移动 python 脚本,这些脚本是我的任务,从已安装的plugins
文件夹中移出它们自己的 docker 容器。
To test this I have created a simple "Hello World:" example-script:为了对此进行测试,我创建了一个简单的“Hello World:”示例脚本:
import numpy as np
def main():
print(f'Hello World')
print(np.random.random((3, 3)))
if __name__ == '__main__':
main()
Which together with the following simple docker-file连同以下简单的 docker-file
FROM python:3.9
ADD main.py .
RUN pip install numpy
CMD ["python", "./main.py"]
I can create my image with docker build -t docker-test-image.
我可以使用docker build -t docker-test-image.
Running this from the CLI with docker run --name docker-test-container docker-test-image
will give me the expected output:使用docker run --name docker-test-container docker-test-image
从 CLI 运行它会给我预期的 output:
Hello World
[[0.20923763 0.25415024 0.95603957]
[0.01320074 0.58392589 0.24175036]
[0.06431375 0.87276564 0.9912474 ]]
So far so good, however if I trigger the DAG with my docker-operator it fails and I get a several到目前为止一切顺利,但是如果我用我的 docker-operator 触发 DAG,它就会失败,我会得到几个
FileNotFoundError: [Errno 2] No such file or directory errors in the logs. FileNotFoundError: [Errno 2] 日志中没有此类文件或目录错误。
My DAG script looks like this:我的 DAG 脚本如下所示:
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import timedelta
with DAG(
dag_id= 'docker_test_dag',
description='Testing the docker operator',
schedule_interval=None,
start_date=days_ago(2),
catchup=False,
tags=['docker_test'],
default_args={
'owner': 'airflow',
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 0,
'depends_on_past': False,
'retry_delay': timedelta(minutes=5)
}
) as dag:
docker_test_task = DockerOperator(
task_id='docker_test_task',
image='docker-test-image',
api_version='auto',
auto_remove=True,
mount_tmp_dir=False,
container_name='docker-test-container',
command='echo "this is a test message shown from within the container',
docker_url='unix://var/run/docker.sock',
network_mode='bridge'
)
docker_test_task
After digging around a bit, I assumed that this is a Docker-in-Docker issue, the most likely solution was found in this tutorial .仔细研究后,我认为这是一个 Docker-in-Docker 问题,最有可能的解决方案是在本教程中找到的。 I added - /var/run/docker.sock:/var/run/docker.sock
to the volumes section in my airflow docker-compose file.我将- /var/run/docker.sock:/var/run/docker.sock
添加到我的 airflow docker-compose 文件中的卷部分。 No the DAG still fails with the following log error:不,DAG 仍然失败,并出现以下日志错误:
Traceback (most recent call last): File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info() 2 File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.7/http/client.py", line 1281, in request self._send_request(method, 88追溯(最近一次通话最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py”,第 450 行,在发送超时=超时文件“/home/airflow/ .local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info() 2 File "/home/ airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py”,第 550 行,递增 raise six.reraise(type(error), error, _stacktrace) File “/home/airflow/ .local/lib/python3.7/site-packages/urllib3/packages/six.py”,第 769 行,在 reraise raise value.with_traceback(tb) File “/home/airflow/.local/lib/python3.7/ site-packages/urllib3/connectionpool.py”,第 710 行,在 urlopen chunked=chunked,文件“/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py”,第 398 行,在 _make_request conn.request(method, url, **httplib_request_kw) 文件“/usr/local/lib/python3.7/http/client.py”, line 1281, in request self._send_request(method, 88 410139179088, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/local/lib/python3.7/http/client.py", line 976, in send self.connect() File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 30, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied')) 410139179088, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request self.endheaders(body, encode_chunked=encode_chunked) 文件 "/usr/local/ lib/python3.7/http/client.py", line 1276, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1036 ,在 _send_output self.send(msg) 文件“/usr/local/lib/python3.7/http/client.py”,第 976 行,在发送 self.connect() 文件“/home/airflow/.local/lib /python3.7/site-packages/docker/transport/unixconn.py", 第 30 行,在连接 sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission拒绝'))
During handling of the above exception, another exception occurred:在处理上述异常的过程中,又出现了一个异常:
Traceback (most recent call last): File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version return self.version(api_version=False)["ApiVersion"] File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version return self._result(self._get(url), json=True) File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner return f(self, *args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get return self.get(url, **self._set_request_timeout(kwargs)) File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py",追溯(最近一次调用最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 214 行,在 _retrieve_server_version return self.version(api_version=False )["ApiVersion"] 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py”,第 181 行,版本返回 self._result(self._get(url ), json=True) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py”,第 46 行,在内部返回 f(self, *args, ** kwargs) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 237 行,在 _get return self.get(url, **self._set_request_timeout(kwargs )) 文件“/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py”,第 542 行,在 get return self.request('GET', url, **kwargs) 文件中“/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py”,第 529 行,请求 resp = self.send(prep, **send_kwargs) 文件“/home/airflow/ .local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))第 645 行,发送 r = adapter.send(request, **kwargs) 文件“/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py”,第 501 行,发送 raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('连接中止', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:在处理上述异常的过程中,又出现了一个异常:
Traceback (most recent call last): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 360, in execute self.cli = self._get_cli() File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 390, in _get_cli return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config) File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in init self._version = self._retrieve_server_version() File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version f'Error while fetching server API version: {e}' docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))追溯(最近一次通话最后一次):文件“/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py”,第 360 行,在执行 self.cli = self._get_cli() 文件“/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py”,第 390 行,在 _get_cli return APIClient(base_url=self. docker_url, version=self.api_version, tls=tls_config) 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”, line 197, in init self._version = self._retrieve_server_version() 文件“/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py”,第 222 行,在 _retrieve_server_version f'Error while fetching server API version: {e }' docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
How is this supposed to be done?这应该怎么做?
Found the answer at last:终于找到答案了:
In the docker-compose file instead of在 docker-compose 文件中,而不是
- /var/run/docker.sock:/var/run/docker.sock
use this on a windows machine:在 windows 机器上使用它:
- //var/run/docker.sock:/var/run/docker.sock
Small change, big effect I guess..小变化,大效果我猜..
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.