Cron activate virtualenv and run multiple python scripts from shell script

Question

I need a cron to run a single shell script. This shell script should activate the python virtualenv and then execute python scripts. This is my cron:

0 0 * * 1 /home/ubuntu/crontab_weekly.sh

This is my crontab_weekly.sh file:

cd /home/ubuntu/virtualenvironment/scripts \
&& source /home/ubuntu/venv3.8/bin/activate \
&& python script1.py \
& python script2.py \
& python script3.py \
& python script4.py \

The idea is to enter the directory where the scripts are hosted, then activate the venv, and only then start executing the scripts. The scripts should be executed in parallel . But in my case only script1.py is executed and the following scripts are not executed. Where is my problem?

Answer 1

Remember that & means to run the entire previous command asynchronously. This includes anything before a && . Commands that run asynchronously run in separate processes.

To take a simplified example of your problem, let's say we asynchronously change directories, run pwd , and asynchronously run pwd again.

#!/bin/sh

cd / && \
pwd \
& pwd

On my computer, this outputs:

/home/nick
/

The cd / was meant to affect both pwd calls, but it only affected the first one, because the second one runs in a different process. (They also printed out of order in this case, the second one first.)

So, how can you write this script in a more robust fashion?

First, I would turn on strict error handling with -e . This exits as soon as any (non-asynchronous) command returns a non-zero exit code. Second, I would avoid the use of && , because strict error handling deals with this. Third, I would use wait at the end to make sure the script doesn't exit until all of the sub-scripts have exited.

#!/bin/sh
set -e

cd /

pwd &
pwd &

wait

The general idea is that you turn on strict error handling, do all of your setup in a synchronous fashion, then launch your four scripts asynchronously, and wait for all to finish.

To apply this to your program:

#!/bin/sh
set -e
cd /home/ubuntu/virtualenvironment/scripts
source /home/ubuntu/venv3.8/bin/activate

python script1.py &
python script2.py &
python script3.py &
python script4.py &

wait