Replace Amazon EC2 SSH hostnames (text file) from terminal output

Question

I have deployed an Amazon EC2 cluster of 3 Ubuntu machines (2 of them make up the cluster and the last one is just a client who submits jobs and manages their storage). I connect to all of them via password-less SSH.

What happens is that every time I restart these machines they get new public hostnames from Amazon which I want to replace in my SSH configuration file located in ~/.ssh/config

So far, I figured out a way to get their names and hostnames using Amazon CLI with the following command at my local machine (CentOS 7):

aws ec2 describe-instances --query "Reservations[*].Instances[*].[PublicDnsName,Tags]" --output=text | grep -vwE "None"

This prints something like

ec2-XX-XX-XXX-XXX.us-east-2.compute.amazonaws.com
Name    datanode1
ec2-YY-YY-YYY-YYY.us-east-2.compute.amazonaws.com
Name    namenode
ec2-ZZ-ZZ-ZZZ-ZZZ.us-east-2.compute.amazonaws.com
Name    client

ie the hostname, a new line, the corresponding name and so on. The IP fields above like XX-XX-XXX-XXX and so on, are basically 4 hyphen separated numbers of 2 or 3 digits. The grep command I have simply removes the last useless line. Now I want to find a way to replace these hostnames to the SSH configuration file or maybe regenerate it, which looks like

Host namenode
  HostName ec2-YY-YY-YYY-YYY.us-east-2.compute.amazonaws.com
  User ubuntu
  IdentityFile ~/.ssh/mykey.pem

Host datanode1
  HostName ec2-XX-XX-XXX-XX.us-east-2.compute.amazonaws.com
  User ubuntu
  IdentityFile ~/.ssh/mykey.pem

Host client
  HostName ec2-ZZ-ZZ-ZZZ-ZZZ.us-east-2.compute.amazonaws.com
  User ubuntu
  IdentityFile ~/.ssh/mykey.pem

Please note that I don't know how the Amazon CLI command sorts the output. But of course, I can change the order of the machines in my SSH file or maybe it is a good idea to delete it and recreate it.

Answer 1

Below is what I finally figured out and it works. This is Bash script you can just save as .sh file like script.sh and execute. If it can't run simply do chmod +x script.sh . I have added comments to clarify what I am doing.

#Ask Amazon CLI for your hostnames, remove the last line, replace the "Name\t" with "", combine every 2 consecutive lines and save to a txt file
aws ec2 describe-instances --query "Reservations[*].Instances[*].[PublicDnsName,Tags]" --output=text | grep -vwE "None" | sed 's/Name\t//g' | sed 'N;s/\n/ /' > 'ec2instances.txt';

#Change the following variables based on your cluster
publicKey="mykey.pem";
username="ubuntu";

#Remove any preexisting SSH configuration file
rm config
touch config

while read line
do
    #Read the line, keep the 1st word and save it as the public DNS
    publicDns=$(echo "$line" | cut -d " " -f1);

    #Read the line, keep the 2nd word and save it as the hostname you will be using locally to connect to your Amazon EC2
    instanceHostname=$(echo "$line" | cut -d " " -f2);

    #OK, we are now ready to store to SSH known hosts
    sshEntry="Host $instanceHostname\n";
    sshEntry="$sshEntry HostName $publicDns\n";
    sshEntry="$sshEntry User $username\n";
    sshEntry="$sshEntry IdentityFile ~/.ssh/$publicKey\n";

    #Attach to the EOF, '-e' enables interpretation of backslash escapes
    echo -e "$sshEntry" >> config

#Below is the txt file you will be traversing in the loop
done < ec2instances.txt

#Done
rm ~/.ssh/config
mv config ~/.ssh/config
rm ec2instances.txt