Running parallel commands in bash

Question

I have a situation where I have a directory "batches" containing several batch files:

one.txt
two.txt
...
seventy.txt

Each of these files needs to be processed by a python script as:

python processor.py --inputFile=batches/one.txt

My current implementation is as such:

for f in batches/$f
do
python processor.py --inputFile=batches/$f
done

I have hundreds of batches, so running all of them in parallel as

python processor.py --inputFile=batches/$f &

Is not feasible.

However,I think that running ~10 at a time shouldn't be a problem.

I'm aware that the syntax

{
python processor.py --inputFile=batches/batchOne.txt
python processor.py --inputFile=batches/batchTwo.txt
} &
{
python processor.py --inputFile=batches/batchThree.txt
python processor.py --inputFile=batches/batchFour.txt
}

Should give me a result similar to the one I wanted. However, are there any better solutions? Basically, given a command template, in my case

python processor.py --inputFile=batches/$1

And a list of batches, I'd like to control how many get executed at the same time.

I'm working on Ubuntu Linux.

Answer 1

Try doing this to run 10 // executions :

parallel -j 10 command_line    

to install it

sudo apt-get install parallel

Answer 2

parallel is a great tool, but not always you have option to install additional packages on a system. You can emulate parallel with bash jobs .

Here is a small example:

#!/usr/bin/env bash

for FILE in /tmp/*.sh;
do
    # count only running jobs. 
    JOBS=$(jobs -r | wc -l)
    while [[ ${JOBS} -ge 3 ]];
    do
        echo "RUNNING JOBS = ${JOBS} => WAIT"
        sleep 5 # too much, just for demo
    done
    echo "APPEND ${FILE} TO JOBS QUEUE [JOBS: ${JOBS}]"
    bash ${FILE} &
done

exit 0

Test:

$ grep '' /tmp/file*.sh
/tmp/file01.sh:sleep 8
/tmp/file02.sh:sleep 10
/tmp/file03.sh:sleep 5
/tmp/file04.sh:sleep 10
/tmp/file05.sh:sleep 8
/tmp/file06.sh:sleep 8

$ ./parallel.sh
APPEND /tmp/file01.sh TO JOBS QUEUE [JOBS: 0]
APPEND /tmp/file02.sh TO JOBS QUEUE [JOBS: 1]
APPEND /tmp/file03.sh TO JOBS QUEUE [JOBS: 2]
RUNNING JOBS = 3 => WAIT
APPEND /tmp/file04.sh TO JOBS QUEUE [JOBS: 2]
RUNNING JOBS = 3 => WAIT
APPEND /tmp/file05.sh TO JOBS QUEUE [JOBS: 1]
APPEND /tmp/file06.sh TO JOBS QUEUE [JOBS: 2]