Apache Airflow DockerOperator 未找到 docker 模块

Question

So I am trying to run a simple dag using the DockerOperator.

I am running the command airflow webserver in my conda environment with the following modules:

Package                Version
---------------------- -------------------
alembic                1.4.2
apache-airflow         1.10.10
apispec                1.3.3
argcomplete            1.11.1
attrs                  19.3.0
Babel                  2.8.0
cached-property        1.5.1
cattrs                 0.9.2
certifi                2020.4.5.1
chardet                3.0.4
click                  7.1.2
colorama               0.4.3
colorlog               4.0.2
configparser           3.5.3
croniter               0.3.32
defusedxml             0.6.0
dill                   0.3.1.1
dnspython              1.16.0
docker                 3.7.3
docker-pycreds         0.4.0
docutils               0.16
email-validator        1.1.1
Flask                  1.1.2
Flask-Admin            1.5.4
Flask-AppBuilder       2.3.4
Flask-Babel            1.0.0
Flask-Caching          1.3.3
Flask-JWT-Extended     3.24.1
Flask-Login            0.4.1
Flask-OpenID           1.2.5
Flask-SQLAlchemy       2.4.3
flask-swagger          0.2.13
Flask-WTF              0.14.3
funcsigs               1.0.2
future                 0.18.2
graphviz               0.14
gunicorn               19.10.0
idna                   2.9
iso8601                0.1.12
itsdangerous           1.1.0
Jinja2                 2.10.3
json-merge-patch       0.2
jsonschema             3.2.0
lazy-object-proxy      1.5.0
lockfile               0.12.2
Mako                   1.1.3
Markdown               2.6.11
MarkupSafe             1.1.1
marshmallow            2.21.0
marshmallow-enum       1.5.1
marshmallow-sqlalchemy 0.23.1
natsort                7.0.1
numpy                  1.18.5
pandas                 0.25.3
pendulum               1.4.4
pip                    20.0.2
prison                 0.1.3
psutil                 5.7.0
Pygments               2.6.1
PyJWT                  1.7.1
pyrsistent             0.16.0
python-daemon          2.1.2
python-dateutil        2.8.1
python-editor          1.0.4
python3-openid         3.1.0
pytz                   2020.1
pytzdata               2019.3
PyYAML                 5.3.1
requests               2.23.0
setproctitle           1.1.10
setuptools             47.1.1.post20200604
six                    1.15.0
SQLAlchemy             1.3.17
SQLAlchemy-JSONField   0.9.0
SQLAlchemy-Utils       0.36.6
tabulate               0.8.7
tenacity               4.12.0
termcolor              1.1.0
text-unidecode         1.2
thrift                 0.13.0
tzlocal                1.5.1
unicodecsv             0.14.1
urllib3                1.25.9
websocket-client       0.57.0
Werkzeug               0.16.1
wheel                  0.34.2
WTForms                2.3.1
zope.deprecation       4.4.0

when this command executes an error is being picked up in the docker_operator.py file when it tries to import docker .

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/ben.muller/airflow/dags/nba_dag.py", line 5, in <module>
    from airflow.operators.docker_operator import DockerOperator
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/docker_operator.py", line 25, in <module>
    from docker import APIClient, tls
ModuleNotFoundError: No module named 'docker'

I don't understand how this is happening, as if I open a simple python REPL in my terminal and import docker there are no issues.

Also get this error when run locally in the "airflow console on localhost:8080"

Does anyone know how to overcome this issue?

Answer 1

The docker module and Airflow are installed in two different environments. The docker module is installed in a virtual environment and Airflow is installed system wide ( /usr/local/lib/python3.7/site-packages/airflow ).

Either install the docker module system wide or install Airflow in the same virtual environment as the docker module.