如何在bash shell中同时运行多个作业

Question

I would like to run the same bash script multiple times on multiple ID's.

Here is the script that i was talking about:

#!/bin/bash
export PATH=/diag/software/bin:$PATH
cd /diag/cloud/udevisetty/RIL_stuff/Block_01
fq_dir="/diag/cloud/udevisetty/RIL_stuff/Block_01"
map_dir="/diag/cloud/udevisetty/RIL_stuff/mapping/"

for RIL in $(cat /diag/cloud/udevisetty/RIL_stuff/Block_01/test.txt); do
      for lane in 01; do
            perl /diag/cloud/udevisetty/RIL_stuff/Block_01/bwa_tophat_final.pl \
            --threads 12 \
            --out $map_dir/$RIL.$lane \
            --bwa_n 0.04 \
            --bwa_db /diag/home/udevisetty/db/Brapa_sequence_v1.5.fa \
            --fq $fq_dir/$RIL/*fq \
            --fq_id $RIL.$lane \
            --bowtie_db /diag/home/udevisetty/db/Brapa_sequence_v1.5.fa \
            --ref_fa /diag/home/udevisetty/db/Brapa_sequence_v1.5.fa \
            --ref_id Brapa_v1.5 \
            > $map_dir/$RIL.$lane.bwa_tophat.log \
            2> $map_dir/$RIL.$lane.bwa_tophat.err
done;
done

My test file contains the following ID's

RIL_251
RIL_21
RIL_211
RIL_136
RIL_66
RIL_155

But right now all it is doing is going through each ID in test.txt file (since it is a 'for' loop) and then executing the script one at a time. However i would like to run all of the IDs simultaneously on 12 threads each. How can i modify my script to do that?

Answer 1

A contrived exampled illustrating how to do this with 'xargs':

root# cat id.txt | xargs -n 1 -P 3 ./script.sh
id2
id1
id4
id3
id5
id7
id6

Contents of id.txt, notice it's ordered.

root# cat id.txt
id1
id2
id3
id4
id5
id6
id7

Script contents:

root# cat script.sh
#!/bin/bash
echo $1

You can also use GNU Parallel if available here

Answer 2

To summarize:

while read -u 3 -r line; do
  process "$line" &
done 3< input.txt

This follows the BashFAQ #001 best-practice approach for reading lines from the file (using FD 3 to avoid reassigning the descriptor for stdin), and uses the & operator to background each process.