minio ModuleNotFoundError when running Airflow Docker image

Question

I have downloaded the latest Docker image for the Airflow and am able to spin up the instance succesfully. On my local system I have installed minio server using homebrew on my Mac.

I have created a DAG file to upload data to my Minio bucket. I have done a sample upload using python and it is working as expected (using the minio python libraries). On the Airflow server I am seeing the following errors -

ModuleNotFoundError: No module named 'minio'

Can someone pleae help me how can I have the pip3 minio library to the docker container so that this error can be resolved? I am new to containers and would really appreciate a easy guide or link that I can refer to help me with this error.

One of the things I did try is to fiddle with the attribute - _PIP_ADDITIONAL_REQUIREMENTS that comes in the AIRFLOW DOCKER image following this link but to no avail.

I added the values as - minio but didn't work.

Answer 1

you can create a Dockerfile that extend the basic airflow and install your packages.

1. Create Dockerfile

 FROM apache/airflow:2.3.0 USER root RUN apt-get update USER airflow RUN pip install -U pip RUN pip install --no-cache-dir minio # or you can copy requirments.txt and install from it

2. Build your docker

docker build -t my_docker .

3. Run the new docker image (if you are using the docker-compose then change the airflow image to your image)