Can't SSH into Google Cloud VM

Question

I was able to successfully SSH into the Google Cloud VM I had set up yesterday, but today for some reason I can't, and I didn't mess with any of the settings, especially not the Firewall settings. It keeps giving me these errors now:

Connection via Cloud Identity-Aware Proxy Failed Code: 4003 Reason: failed to connect to backend You may be able to connect without using the Cloud Identity-Aware Proxy.

Then when I click on "Connect without Identity-Aware Proxy" I get the following error:

Connection Failed We are unable to connect to the VM on port 22. Learn more about possible causes of this issue.

I don't know what happened. Yesterday it was working fine and now it's not.

Answer 1

Beased on these error messages, I guess that your project has Identity-Aware Proxy (IAP) enabled , which sometimes may affect the ability to SSH into an instance, depending on the configuration.

In order to rule out this, you may try the following:

• Create the firewall rules for allowing IAP to connect to your instances
• Grant the necessary permissions to use IAP
• Tunnel the SSH connection through IAP

Answer 2

At first, try to disable Cloud Identity-Aware Proxy and connect to the VM instance via web Console.

After that, check logs:

• Go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM -> at the VM instance details find section Logs and click on Serial port 1 (console)
• Reboot your VM instance.
• Check full boot log for any errors or/and warnings.

If your VM instance doesn't start up verify that your disk has a valid file system and a valid master boot record (MBR) by following the documentation General troubleshooting .

If you found errors/warning related to disk space you can try to resize it accordingly to the documentation Resizing a zonal persistent disk , also accordingly to the article Recovering an inaccessible instance or a full boot disk :

If an instance is completely out of disk space or if it is not running a Linux guest environment, then automatically resizing your root filesystem isn't possible, even after you've increased the size of the persistent disk that backs it. If you can't connect to your instance, or your boot disk is full and you can't resize it, you must create a new instance and recreate the boot disk from a snapshot to resize it.

Otherwise try get access to your VM instance via serial console:

• Enable serial console connection with gcloud command:

   gcloud compute instances add-metadata NAME_OF_YOUR_VM_INSTANCE \ --metadata serial-port-enable=TRUE

  or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and check Enable connecting to serial ports

• Create temporary user and password to login: shutdown your VM and set a startup script by adding at the section Custom metadata key startup-script and value:

   useradd --groups google_sudoers tempuser echo "tempuser:password" | chpasswd

  and then start your VM.

• Connect to your VM via serial port with gcloud command:

   gcloud compute connect-to-serial-port NAME_OF_YOUR_VM_INSTANCE

  or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> and click on Connect to serial console

• Check what went wrong.

• Disable access via serial port with gcloud command:

   gcloud compute instances add-metadata NAME_OF_YOUR_VM_INSTANCE \ --metadata serial-port-enable=FALSE

  or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and uncheck Enable connecting to serial ports . Keep in mind that accordingly to the documentation Interacting with the serial console :

  Caution : The interactive serial console does not support IP-based access restrictions such as IP whitelists. If you enable the interactive serial console on an instance, clients can attempt to connect to that instance from any IP address. Anybody can connect to that instance if they know the correct SSH key, username, project ID, zone, and instance name. Use firewall rules to control access to your network and specific ports.

If you weren't able to connect via serial console, try follow the documentation Troubleshooting SSH section Inspect the VM instance without shutting it down and inspect the disk of your VM on another VM. Same way you can transfer your data to another working VM instance.

Answer 3

I had had the same issue while running composer update.

In my case an rebooting of the VM-Instance has solved it.