Shell script loop through file, ssh into remote server and copy files continue on fail

Question

Given the following file, I cannot figure out how to continue to iterate through the lines one by one without it exiting on first failure of the inner most remote ssh command.

The basic premise is this script ssh's to a remote server defined as INPUT_FILE as USER and copies a bunch of files off to a remote location for backup.. the actual backup script is in backup.sh . which controller.sh executes.

My problem is that if the backup.sh throws an error, something like a permission denied for cp the whole while loop finishs and it won't connect to any of the remaing servers in my INPUT_FILE . I know this because I have purposely put in place permissions on various servers to test this scenario.

How can i iterate through the files, and wait for the first ssh command to complete and force it to go to the next line without exiting.

controller.sh

INPUT_FILE=$1
USER=$2
SCRIPT_DIR=/opt/sapdownloads/MikesUtilities/backups
BACKUP_DIR=$SCRIPT_DIR/data
DATE=`date +%m%d%y`
LOG_FILE=$SCRIPT_DIR/logs/backup.controller.log


##
# Create The Log dir and log files
##
function log(){
   echo -e ${*}
   echo -e "[${USER}][`date`] - ${*}" >> ${LOG_FILE}
}


#Iterate through the input file line by line

cat $INPUT_FILE | while read l;do {
   #Split the line by spaces into an array
   line=($l)
   #make sure server is lower case
   SERVER=`echo ${line[1]} | tr [:lower:] [:upper:]`
   #make sure tier is uppercase
   TIER=`echo ${line[0]} | tr [:upper:] [:lower:]`
   log "###################################"
   log "#  Tier: $TIER | SERVER: $SERVER  #"
   log "###################################"
   log "Making Sure we can login to the Server"
   CHK=`ssh -t -q -o "BatchMode yes" -o "ConnectTimeout 5" -l $USER $SERVER "echo success"`;
   if [ "success" = $CHK ] >/dev/null 2>&1
   then
      CPDIR=$BACKUP_DIR/$TIER/$SERVER/$DATE
      log "Creating new backup directories in $CPDIR"
      #make the new backup directory in $BACKUP_DIR
      mkdir -p $BACKUP_DIR/$TIER/$SERVER/$DATE/etc
      mkdir -p $BACKUP_DIR/$TIER/$SERVER/$DATE/home


      log "Executing Remote Backup Script"
      ret=`ssh -n -t -q -o "BatchMode yes" -o "ConnectTimeout 5" -l $USER $SERVER "/bin/bash $SCRIPT_DIR/backup.sh $CPDIR; echo $?"`
      echo "return: [$ret]"
      if [ $ret -eq 0 ]; then
         echo "Success. Aborting"
         break
      fi
   else
      printf "${i}\tUnable to connect to host\n" >> $ERRORS;
   fi
} </dev/null; done

backup.sh

#!/bin/bash
CPDIR=$1
DATE=`date +%m%d%y`
SCRIPT_DIR=/opt/sapdownloads/MikesUtilities/backups
LOG_FILE=$SCRIPT_DIR/logs/backup.log
SIDADM_HOME=/home/${sapsid}adm
ORASID_HOME=/home/ora${sapsid}
ORACLE_HOME=/home/oracle
cpErrors=()
message=""
SENDTO="joe@example.com"
SENDFROM="team@example.com"

##
# Create The Log dir and log files
##
function log(){
   #echo -e ${*}
   echo -e "[${USER}][`date`] - ${*}" >> ${LOG_FILE}
}

function compileMessage(){
   message="Unable to CP one or more files from the server $HOSTNAME \n\n"
   for i in "${cpErrors[@]}"
   do
      message+="${i}\n"
   done
}

function sendMail(){
   compileMessage

   echo -e $message | mail -s "OS Backup Errors" -r $SENDFROM $SENDTO
}

cd /etc
#copy fstab
if [ -d "oratab" ]; then
   log "Copying oratab to $CPDIR/etc"
   cp oratab $CPDIR/etc
fi
#Copy fstab
if [ -d "fstab" ]; then
   log "Copying fstab to $CPDIR/etc"
   cp fstab $CPDIR/etc
fi

#make sure that folder exists first
if [ -d "$ORASID_HOME" ]; then

   #Test to make sure we can access the folder
   if [ -w "$ORASID_HOME" ]; then
      log "Copying $ORASID_HOME to $CPDIR/home"
      cp -rf $ORASID_HOME $CPDIR/home
   else
      #if we cant access that folder, send an email
      log "Unable to cp $ORASID_HOME to $CPDIR/home"
      cpErrors+=($ORASID_HOME)
   fi

fi

#make sure that folder exists first
if [ -d "$SIDADM_HOME" ]; then

   #Test to make sure we can access the folder
   if [ -w "$SIDADM_HOME" ]; then
      log "Copying $SIDADM_HOME to $CPDIR/home"
      cp -rf $SIDADM_HOME $CPDIR/home
   else
      #if we cant access that folder, send an email
      log "Unable to cp $SIDADM_HOME to $CPDIR/home"
      cpErrors+=($SIDADM_HOME)
   fi

fi

#make sure that folder exists first
if [ -d "$ORACLE_HOME" ]; then

   #Test to make sure we can access the folder
   if [ -w "$ORACLE_HOME" ]; then
      log "Copying $ORACLE_HOME to $CPDIR/home"
      cp -rf $ORACLE_HOME $CPDIR/home
   else
      #if we cant access that folder, send an email
      log "Unable to cp $ORACLE_HOME to $CPDIR/home"
      cpErrors+=($ORACLE_HOME)
   fi

fi


if [ ${#cpErrors[@]} != 0 ]; then
   log "Sending Email of cp errors"
   sendMail
fi

Answer 1

As you probably have guessed, ssh is eating your input from the while loop, and that's why it's failing. Actually I'm not 100% sure why -n is not working in this case, but we can work around it.

Since I don't have your backup scripts (nor the general setup), I simplified the script quite a bit, but I think the general idea is same. Go through input file that server information, and try to find a server we can login to, and then see if we can successfully execute a script there

I call a simple script on a remote machine that waited for couple of seconds, and returned an exit code . You should add some exit codes to your Backup.sh , so that you can check what was returned from the remote command, and then act on it.

So with adding curly braces to the while (and </dev/null ), and retrieve exit code from your remote script, I was able to go through the list until a successful remote script execution was done:

cat $INPUT_FILE | while read l;do {

SERVER=($l)
CHK=`ssh -t -q -o "BatchMode yes" -o "ConnectTimeout 5" -l $USER $SERVER "echo success"`;

if [ "success" = $CHK ] >/dev/null 2>&1
then
  ret=`ssh -n -t -q -o "BatchMode yes" -o "ConnectTimeout 5" -l $USER $SERVER '/home/username/timer.sh testvar;echo $?'`
  echo "return: [$ret]"
  if [ $ret -eq 0 ]; then
       echo "Success. Aborting"
       break
  fi
else
  printf "${i}\tUnable to connect to host\n";
fi
} < /dev/null; done

Hope it helps