A bash shell program to wait for all servers in a list to start, then run startup scripts

Question

I'm trying to automate the startup (after reboot) of an application that runs on a grid of 12 Linux 7 servers.

The 12 servers get rebooted in random order.

All the servers need to be running before I can start up the application that resides on the 12 servers.

What I'd like to do is to test that all 12 Linux 7 servers are up and then when all 12 are up, I'd proceed with the startup sequence of commands.

All 12 server are set up with ssh keys.

cat serverlist.txt 10.0.0.6 10.0.0.7 10.0.0.8 10.0.0.9 ... 10.0.0.18

I want to ping a server and then wait until the ping is successful, then move to the next IP address.

My apologies for my question.

How to code this?

Read in the first line from IP list file. Ping first IP until success, then ping the other IP addresses (one at a time) until the success of all 12.

Then, run commands to start the application on the grid of 12 servers.

Question: How to code this in the bash shell.

Answer 1

The inner loop can be as simple as

while ! ssh "${connection_string}" -o ConnectTimeout=5 true
do
    sleep 0.5
done

This runs a trivial command, and waits for 0.5 seconds between retries.

Answer 2

for i in `cat /home/Startup/serverlist.txt`
do
    ssh ${i} -o ConnectTimeout=5 true
    while test $? -gt 0
    do
        ssh ${i} -o ConnectTimeout=5 true
    done
done
exit  

If ssh works the server is alive, plus the use of the timeout is a better way to speed up the script execution. If the script get finish that means all the servers are up and responding.