How can I make an AWS EC2 allow ssh connections regardless of the EBS in /dev/sda1?

Question

I have an app that has been successfully running on EC2 for a few years. The system is Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1032-aws x86_64).

It's a small and simple app with low traffic. I had never made any changes to the server itself until today. I wanted to deal with the X packages can be updated. message, so I ran:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

Then I ran sudo reboot . Once rebooted, the app runs perfectly. I can access it as normal via the public URL and look at things, including db (postgresql directly on the server) data with no issues or surprises.

But, when I tried to ssh into the machine again, I couldn't. I do ssh -i "key.pem" -vvv ubuntu@<IP> and get:

debug1: Connecting to <IP> [<IP>] port 22.
debug1: connect to address <IP> port 22: Operation timed out
ssh: connect to host <IP> port 22: Operation timed out

No changes were made to security groups. Also, it's such a small project, that I never setup EC2 Instance Connect or anything like that.

I had the thought of launching a new EC2 and just switching the EBS volumes, thinking EBS would bring the app and data, while the instance itself would have configs and permissions.

I do not understand much about this (clearly), and was surprised to learn that the EBS volume itself seems to be the problem and hold all the cards.

I can switch EBS volumes back and forth between the two EC2 instances. At any given time, whichever one has the newest (and therefore blank) EBS volume attached at /dev/sda1 allows SSH but surely does not run the app. And, vice-versa: Whichever EC2 instance has the original EBS volume runs the app perfectly but keeps me locked out of ssh.

In this scenario, the question is: How can I make one of the EC2 instances bypass this EBS issue and make its own decision about allowing me to connect with ssh?

Or: What is the obvious and/or silly thing I'm missing here?

PS: I do have elastic IP going for all of this, so it doesn't seem like DNS would be the source of the problem.

Answer 1

With John Rotenstein's help, I was able to resolve this.

Here are the core steps:

Phase 1 - Attach and mount additional volume

Per John's comment, it's possible to boot the instance from the "good" volume and then attach and mount the "bad" volume after. This allowed me to explore files and look for issues.

AWS panel

1. Attach volume to EC2 instance as root by using /dev/sda1 for name
2. Start the EC2 instance
3. Attach the other volume after instance has booted

Terminal

1. SSH into the server

2. See root volume information:

~$ df -hT /dev/xvda1

3. Check for mounted volumes:

~$ lsblk

4. See additional volume information:

~$ df -hT /dev/xvdf1

5. Switch to root user:

~$ sudo su -

6. Make a directory to be the mount path:

~$ mkdir /addvol

7. Mount the additional volume to the path:

~$ mount /dev/xvdf1 /addvol

8. Check additional volume contents:

~$ ls -la /addvol/home/ubuntu

Now I could see and navigate the additional volume's contents, finding config files, looking at authorized_keys, file permissions, etc.

This article from AWS helped a lot to get me here.

After finally getting to this point, I could not find any problems with the keys, or permissions, etc. John pointed me to this article dealing with Ubuntu's firewall things .

Phase 2 - Dealing with the firewall

I ran some commands from the article and tried to understand how they worked.

Once I grasped it a little, I decided to use an existing reboot script I have on the volume to ensure the firewall was ok with SSH connections.

I updated my existing custom reboot script, adding the following lines:

sudo ufw allow ssh
sudo ufw allow 22
sudo ufw disable
sudo ufw --force enable

Basically it calls to allow for ssh twice, once by name and then by port. I'm a newbie on this stuff and just went for the overkill.

Then it disables and enables the firewall to ensure it runs with these news things configured.

Because sudo ufw enable requires an interaction, I chose to use sudo ufw --force enable .

Phase 3 - Testing and using it!

After the script update, I exited the server.

AWS panel:

1. Stop the EC2 instance
2. Detach one volume from the instance
3. Detach the other volume from the instance
4. Reattach the "bad" volume, this time as root
5. Start the EC2 instance

Terminal:

1. SSH into the instance - Voila!

NOTE: Before truly working 100%, my computer complained about the known_hosts thing. The server key must have changed on the update/upgrade and/or after all of the volume changes. I don't think having to confirm hosts is a big deal, so I just usually clear all of the contents in my local .ssh/known_hosts file. If you prefer to be specific, you can find the server's information on there specifically and delete only the relevant lines.