Docker Multi-stage Builds of miniconda environments and Python

Question

i want to use multi stage build in docker. the first stage is to install faiss library, and the second stage is to install the main images. here is my script

FROM conda/miniconda3 as install_faiss

WORKDIR /app

RUN conda install -c conda-forge faiss-cpu

FROM python:3.7.11 as base

WORKDIR /app

COPY requirements.txt ./app/requirements.txt
COPY data_train.csv ./app/data_train.csv
COPY train.py ./app/train.py

RUN pip3 install -qr ./app/requirements.txt
COPY --from=install_faiss ./app ./app
RUN python3 ./app/train.py

but after that in may file train.py error getting like cannot import faiss, so i assume that module faiss not copy in the second image. any help? thanks

Answer 1

I'm not sure how you expect this to work?

The conda install in the first stage runs in the /app working directory, but it will install the faiss-cpu package in conda's site-packages directory. I don't know how the conda/miniconda3 docker image is set up, but I'm quite sure it does not simply install packages in the current working directory.

In the second image you have a python 3.7.11 installation where you installed your requirements.txt. These requirements will be installed in python 3.7.11's site-packages directory. That's also where you want to install faiss-cpu.

There's probably a way to figure out where the correct site-packages directories are in both the miniconda and python base images and then use the correct COPY --from command.

But why do you use conda to install the first dependency and pip for the others? Can't you use a single image and do:

pip3 install faiss-cpu