Running qsub with anaconda environment

Question

I have a program that usually runs inside a conda environmet in Linux, because I use it to manage my libraries, with this instructions:

source activate my_environment
python hello_world.py

How can I run hello_world.py in a high computer that works with PBS. Instructions explains to run adapting the code script.sh , shown below, and calling with the instruction qsub .

# script.sh
#!/bin/sh
#PBS -S /bin/sh
#PBS -N job_example
#PBS -l select=24
#PBS -j oe
cd $PBS_O_WORKDIR
mpiexec ./programa_mpi

How do I run hello_world.py with qsub using my anaconda environment?

Answer 1

You'll need to load the Python module before activating your environment and before running your script.

module load python3

cd $PBS_O_WORKDIR
source activate my_environment

mpiexec python hello_world.py

Check the documentation for your institution regarding their Python modules. At my institution, Anaconda was the environment module for Python3, so you could load it as I have shown.

Answer 2

Unless it is in the environment by default, one also needs to "load" conda as well, inside the SGE(qsub) script (similar thing would hold for a slurm script too I assume). For example I had installed conda to the directory seen below in the SGE script, so I export the path (if conda is installed as a module on an HPCC, then simply load that instead):

#!/bin/bash
#$ -q compute
#$ -l compute
#$ -cwd
#$ -N name
#$ -j yes

export PATH=$HOME/miniconda3/bin:$PATH

source activate my_environment

environment function code...

Answer 3

If your really need your conda environment you can activate it in the script

source ~/.bashrc #configures your shell to use conda activate
conda activate your_env_name