With OS login, how to gcloud ssh from one GCE into another GCE?

Question

I have two GCEs, both has OS login enabled:

1) a_vm provisioned with service account a_svc

2) b_vm provisioned with service account b_svc

a_svc has OS login access to b_vm. a_svc has roles/compute.osAdminLogin at project level and roles/iam.serviceAccountUser at b_svc level.

What I tried:

• Ssh into a_vm and gcloud compute ssh a_vm , then gcloud compute ssh b_vm

What happened:

It times out consistently. Requests gcloud made before it hangs based on --log-http :

uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/a_svc@fredzqm-terraform-5.iam.gserviceaccount.com/?recursive=True
uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/ca_svc@my_project.iam.gserviceaccount.com/token
uri: https://compute.googleapis.com/batch/compute/v1
uri: https://compute.googleapis.com/batch/compute/v1
uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/a_svc@my_project.iam.gserviceaccount.com/?recursive=True
uri: https://oslogin.googleapis.com/v1/users/a_svc@my_project.gserviceaccount.com/loginProfile?projectId=fredzqm-terraform-5&alt=json

What I expects:

Based on https://cloud.google.com/compute/docs/instances/connecting-advanced#sa_ssh_manual , gcloud should pick up the default application credential (a_svc) and use it to get access to b_vm through OS login.

What works:

• I have verified that a_svc does have ssh access to b_vm. This works from my workstation using personal creds:

  gcloud compute ssh b_vm --impersonate-service-account a_svc

• If I add a ssh public key to a_svc, and then use the public key to ssh into b_vm .

  gcloud compute os-login ssh-keys add --key-file

  ssh sa_<id_of_a_svc>@b_vm -i private_key_path

Answer 1

I've tried to reproduce your issue and failed. I was able to connect to service-account-b-instance from service_account_a_instance as the service account and execute commands as that service account.

Have a look at my steps below:

1. create service account service_account_a

2. create VM instance associated with service_account_a_instance :

    $ gcloud compute instances create service-account-a-instance --zone=europe-west3-a --machine-type=n1-standard-1 --service-account=service-account-a@test-prj.iam.gserviceaccount.com --scopes=https://www.googleapis.com/auth/cloud-platform --metadata enable-oslogin=TRUE Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/service-account-a-instance]. NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS service-account-a-instance europe-west3-a n1-standard-1 10.156.0.14 35.XXX.75.XXX RUNNING

3. create service account service_account_b

4. create VM instance service_account_b_instance associated with service_account_b :

    $ gcloud compute instances create service-account-b-instance --zone=europe-west3-a --machine-type=n1-standard-1 --service-account=service-account-b@test-prj.iam.gserviceaccount.com --scopes=https://www.googleapis.com/auth/cloud-platform --metadata enable-oslogin=TRUE Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/service-account-b-instance]. NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS service-account-b-instance europe-west3-a n1-standard-1 10.156.0.16 35.XXX.255.XXX RUNNING

5. generate ssh keys:

    $ ssh-keygen Generating public/private rsa key pair.

6. import ssh key:

    $ gcloud compute os-login ssh-keys add --key-file id_rsa.pub

7. connect to instance service-account-a-instance :

    $ gcloud compute ssh service-account-a-instance Linux service-account-a-instance 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Mon Mar 2 14:34:51 2020 from 104.132.189.65 user_domain_com@service-account-a-instance:~$

8. connect to instance service-account-b-instance from service-account-a-instance :

    user_domain_com@service-account-a-instance:~$ gcloud compute ssh service-account-b-instance --project test-prj --zone europe-west3-a ... ssh: connect to host 35.242.255.44 port 22: Connection timed out ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

9. enable ssh connectivity at VPC network -> Firewall and try again:

    user_domain_com@service-account-a-instance:~$ gcloud compute ssh service-account-b-instance --project test-prj --zone europe-west3-a Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

10. add role roles/iam.serviceAccountUser to service-account-a and try again:

     user_domain_com@service-account-a-instance:~$ gcloud compute ssh service-account-b-instance --project test-prj --zone europe-west3-a Linux service-account-b-instance 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Mon Mar 2 16:52:28 2020 from 35.198.75.226 sa_116762935227008431464@service-account-b-instance:~$ sa_116762935227008431464@service-account-b-instance:~$ uname -a Linux service-account-b-instance 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 GNU/Linux

    and finally it works.