How to add local python files as a library to python virtual environment the same way as pip install to Airflow Docker

Question

• How to install my own python package to python virtual environment?
• Final goal would be to add that package to a Airflow Docker environment.
• My Dockerfile:

FROM apache/airflow:latest-python3.8
COPY requirements.txt .
RUN pip install -r requirements.txt

Answer 1

First, check the answer here https://stackoverflow.com/a/56483981/11609051 for the installation of your package to the venv.

After that you need to extend your docker image. For this your Dockerfile.txt should be similar to:

FROM apache/airflow:2.4.0
WORKDIR /Users/Desktop/tools/airflow2.4-lite
COPY requirements.txt /requirements.txt
RUN pip install --user --upgrade pip
RUN pip install --no-cache-dir --user -r /requirements.txt 

then you need to run command:

docker build . -f Dockerfile.txt 

However there are some points that needs to be taken into account.

• Your requirements.txt file should be in your Airflow Docker project file.
• Any of your packages in your venv shouldn't have the dependency of Python greater than of your Airflow's which is Python 3.8 nor they shouldn't depend on a package that uses a dependent rely on greater than Python 3.8.
• For the above you can use the Homebrew's Python@3.8 and you can install the packages in site-packages directory with: pip3.8 install <package-name> command.