Airflow Docker - 没有名为“pyodbc”的模块

Question

I have a Python script which uses Pyodbc to work with SQL Server, I'm trying to run this script with an Airflow DAG:

auction_task = PythonOperator(
  task_id='auction_etl',
  python_callable=get_auction_data,
  dag=dag
)

get_auction_data function in another file:

import json
import requests
import datetime
import pyodbc
import time

def get_auction_data(config):
    return retrieve_from_api(config)

But I keep getting this error in the UI:

Broken DAG: [/opt/airflow/dags/auctionds/etl/auction_etl.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/dags/auctionds/etl/auction_etl.py", line 9, in <module>
    import pyodbc
ModuleNotFoundError: No module named 'pyodbc'

Airflow is running inside a Docker container, I tried to manually install pyodbc in the worker container with no success. Also, if I run python command inside the worker container and run 'import pyodbc' it works with no problem. I don't know what I'm missing. Thanks!

Answer 1

You need to create a custom image with pyodbc added. You can see some examples on how to do it in https://airflow.apache.org/docs/docker-stack/build.html#how-to-build-your-own-image including example on how to add new python packages.

Then you have to use that custom image in your docker deployment (however you deploy the container - might be docker container or K8S for example). You just need to make your image available to the deployment and make sure your new image is used by it.