Bash script for running parallel commands using input from a file

Question

I am trying to make a shell script which reads a configuration file and executes commands for each line in parallel.

For example, I have IPs.cfg, which can contain a variable amount of IPs. It could be one or several

IPs.cfg

145.x.x.x
176.x.x.x
192.x.x.x

I want to read the file and then execute a command for each line at the same time... for instance :

scp test.iso root@$IP1:/tmp &
scp test.iso root@$IP2:/tmp &
scp test.iso root@$IP3:/tmp &
wait

The way I'm thinking this is that I store the IPs into an array

IFS=$'\n' read -d '' -r -a array < IPs.cfg

Then I extract the number of lines from the file and decrease it by 1 since the array starts at 0.

NUMLINES=`cat IPs.cfg | wc -l`
NUMLINES=$((NUMLINES-1))

Now I want to execute the commands all at the same time. It's a variable number of parameters so I can't just manually use scp test.iso root@${array[0]}:/tmp & scp test.iso root@${array[1]}:/tmp & so on . I could use a while loop, but that would mean doing the commands one at a time. I'm also thinking about using recursion, but I have never done that in a bash script.

It might be a silly question, but what are my options here?

Answer 1

您可以利用GNU并行

Answer 2

The loop should look like this:

while read -r ip ; do
   scp test.iso "root@$ip:/tmp" &
done < IPs.conf
wait

Answer 3

with this trick, you can control the number of simultaneous processes running at the same:

cat IPs.conf | xargs -n1 -I{} -P10 scp test.iso "root@{}:/tmp"

Check the -p10 which means to use 10 processes at the same time.