Bash and Conda: Installing non-conda packages in conda environment with executable bash script

Question

I am writing a bash script with the objective of hosting it on a computing cluster. I want the script to create a conda environment for whichever user executes it, so that everyone on our team can quickly set-up the same working environment.

I realize this is a bit overkill for the number of commands necessary but I wanted to practice some bash scripting. Here is my script so far:

#!/bin/bash

# Load anaconda
module load Anaconda/4.2.0

# Create environment
conda create -n ADNI

# Load environment
source activate ADNI

# Install image processing software
pip install med2image

echo 'A working environment named ADNI has been created.'
echo 'Please run `source activate ADNI` to work in it.'

This script creates the environment successfully. However, once I load the environment after running the script, I run conda list to see which packages are loaded within it and get the following output:

(ADNI) MLG-BH0039:ADNI_DeepLearning johnca$ conda list
# packages in environment at /Users/johnca/miniconda3/envs/ADNI:
#

(ADNI) MLG-BH0039:ADNI_DeepLearning johnca$ 

This gives me the impression that the environment has no packages loaded in it. Is this correct? If so, how can I alter the script so that the desired packages successfully install into the specified environment.

Thanks!

Answer 1

I managed to find a better way to automate this process by creating an environment.yml file with all the desired packages. This can include pip packages as well. My file looks like this:

name: ADNI
channels:
- soumith
- defaults
dependencies:
- ca-certificates=2017.08.26=h1d4fec5_0
- certifi=2017.11.5=py36hf29ccca_0
- cffi=1.11.2=py36h2825082_0
- freetype=2.8=hab7d2ae_1
- intel-openmp=2018.0.0=hc7b2577_8
- jpeg=9b=h024ee3a_2
- libffi=3.2.1=hd88cf55_4
- libgcc=7.2.0=h69d50b8_2
- libgcc-ng=7.2.0=h7cc24e2_2
- libgfortran-ng=7.2.0=h9f7466a_2
- libpng=1.6.32=hbd3595f_4
- libstdcxx-ng=7.2.0=h7a57d05_2
- libtiff=4.0.9=h28f6b97_0
- mkl=2018.0.1=h19d6760_4
- numpy=1.13.3=py36ha12f23b_0
- olefile=0.44=py36h79f9f78_0
- openssl=1.0.2n=hb7f436b_0
- pillow=4.2.1=py36h9119f52_0
- pip=9.0.1=py36h6c6f9ce_4
- pycparser=2.18=py36hf9f622e_1
- python=3.6.0=0
- readline=6.2=2
- scipy=1.0.0=py36hbf646e7_0
- setuptools=36.5.0=py36he42e2e1_0
- six=1.11.0=py36h372c433_1
- sqlite=3.13.0=0
- tk=8.5.18=0
- wheel=0.30.0=py36hfd4bba0_1
- xz=5.2.3=h55aa19d_2
- zlib=1.2.11=ha838bed_2
- pytorch=0.2.0=py36hf0d2509_4cu75
- torchvision=0.1.9=py36h7584368_1
- pip:
  - cycler==0.10.0

I can then automate creating the environment by referencing this file, as in:

#!/bin/bash

# Load anaconda
module load Anaconda/4.2.0

# Create environment
conda env create -f adni_env.yml

echo ' '
echo 'A working environment named ADNI has been created or updated.'
echo 'If working on the cadillac server please `module load Anaconda/4.2.0`.'
echo 'Then run `source activate ADNI` to work within the environment.'
echo ' '

I hope this can help anyone in the future who may have similar issues.

Answer 2

The command

conda create -n ADNI

creates an environment with no packages installed, not even Python or pip. Therefore, despite activating the environment, you are still using some other pip that appears on your PATH . You need to install pip or Python into the environment first, either when the environment is created or afterwards with the conda install command

conda create -n ADNI python=3.6

will install Python, which brings along pip when the environment is created or

conda create -n ADNI
conda install -n ADNI python=3.6

will install Python afterwards.

In the best case, you would use conda to install that package. It isn't all that difficult to create a conda package from a pip package and upload it to a channel on Anaconda.org so your team can access it.