How to run ssh connection in parallel while running multiple commands Linux

Question

I am trying to understand best way to run multiple command on a remote host with ssh in a parallel way and Display the output on the screen while squeezing the warning and errors on the screen.

This post is mainly in connection with previous post where KamilCuk answered the POST with some beautiful tricks however i have some more open questions.

Below code works for the given commands, however few of my systems really old and do not understand the nproc command to get the number of processor on the system hence i need to use grep -c processor /proc/cpuinfo which is not working if i do so in the current solution like tmp=$(ssh "$server" bash -c 'grep -c "^processor" /proc/cpuinfo; free -g') .

also would be look to incorporate conditions like if nproc is there then use the file /proc/cpuinfo to count the CPU, this is Just one thing similar to many other. So, what i am looking forward if there are more ways to do it if you have different subsets of command where you need to get values from commands and files .

 #!/bin/bash

read -rsp $'Please Enter password below: ' SSHPASS
export SSHPASS

work() {
   server=$1
   # ONE connection
   tmp=$(ssh "$server" bash -c 'nproc; free -g')
   # parsing later
   cpu_info=$(<<<"$tmp" awk 'NR==1')
   mem_info=$(<<<"$tmp" awk '/Mem:/{printf $2}')
   swap_info=$(<<<"$tmp" awk '/Swap:/{printf $2}')
   # outputting
   printf "%-40s %5s %5s %5s\n" "$server" "$cpu_info" "$mem_info" "$swap_info"
}
export -f work
< /home/user1/mem xargs -P0 -n1 -d'\n' bash -c 'work "$@"' _

I tried Below where the idea is borrowed from But it just prints the SEREVR name and nothing else:

#!/bin/bash
read -rsp $'Please Enter password below: ' SSHPASS
export SSHPASS

for SERVER in '$(cat /home/user1/mem)'
do
sshpass -e ssh -q -t -oStrictHostKeyChecking=no $SERVER << EOF
        cpu_info=$(grep processor /proc/cpuinfo | awk 'NF==3{count++} END {printf count}')
        mem_info=$(free -g | awk /Mem:/{printf' '})
        swap_info=$(free -g | awk /Swap:/{printf' '})
        
        printf "%-40s %5s %5s %5s\n" "$server" "$cpu_info" "$mem_info" "$swap_info"

EOF
done | tee  2>/dev/null

I may have multiple mistakes please excuse me for the same just trying to learn.

EDIT Note:

I Know there will be a definite questions why not use ssh keys or already existed tooling like ansible, salt etc, the reason is only, due to company security policies somewhat we can not use any of them else there is no way re-inventing the wheel, however we are using Ansible and other tooling heavily in the normal environment.

Answer 1

First, this loop won't work:

for SERVER in '$(cat /home/user1/mem)'
do
...
done

because of '', remove '' or better read hosts list in var or an array first:

var=$(cat  /home/user1/mem)
arr=($(cat /home/user1/mem))

and loop over this var(arr):

for SERVER in   $var     ; { ...; }
for SERVER in "${arr[@]}"; { ...; }

Parallel ssh connections can be done like this:

cmd="
    cpu_info=\$(grep processor /proc/cpuinfo | awk 'NF==3{count++} END {printf count}')
    mem_info=\$(free -g | awk /Mem:/{printf' '})
    swap_info=\$(free -g | awk /Swap:/{printf' '})

    printf \"%-40s %5s %5s %5s\n\" \"\$HOSTNAME\" \"\$cpu_info\" \"\$mem_info\" \"\$swap_info\"
"

for SERVER in "${arr[@]}"; {
    sshpass -p "$SSHPASS" ssh -q -t -oStrictHostKeyChecking=no "$SERVER" "$cmd" &
}

Answer 2

Another solution to get these values you need could be:

$ ssh user@foobar "bash -c \"grep -c '^processor' /proc/cpuinfo; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'
user@foobar's password:
4 Mem: 15 2 2 0 10 11 Swap: 15 0 15                                                                                                                                                                                                

It could be good if you tell us what are the command or the tools available in these "old" systems. I have only used to use the commands you have in your script (grep and awk).

For the multiple SSH connections, just introduce the command line into the for loop.

UPDATE: To handle the problem that if the machine does not have command nproc just add an if condition:

if ! command -v nproc &> /dev/null
then
        ssh user@foobar "bash -c \"grep -c '^processor' /proc/cpuinfo; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'

else
        ssh user@foobar "bash -c \"nproc ; free -g\"" | awk '{if (NR==1) {printf $1 " "} if ($1=="Mem:") { for (i=1; i<=NF; i++) printf $i " "}; if ($1=="Swap:") { for (i=1; i<=NF; i++) printf $i " "};}'
fi