ssh to google cloud Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Question

spent more than 10 hours trying to troubleshoot the reason why I cannot ssh to gcloud engine through my mac. I've added my public key to sshKeys metadata. I can ssh to the other instances under the same project. But this one doesnt work.

ssh mxiong@xx.xx.xx.xx -vvv
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/mxiong/.ssh/config
debug1: /Users/mxiong/.ssh/config line 1: Skipping Host block because of negated match for *.*
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "xx.xx.xx.xx" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /Users/mxiong/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/mxiong/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to xx.xx.xx.xx:22 as 'mxiong'
debug3: hostkeys_foreach: reading file "/Users/mxiong/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/mxiong/.ssh/known_hosts:31
debug3: load_hostkeys: loaded 1 keys from xx.xx.xx.xx
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug3: receive packet: type 31
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 1540/3072
debug3: send packet: type 32
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug3: receive packet: type 33
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:JHpcutx43AfJ/xKjlX23IjSvH3gYxSiiHbp17WkcBdc
debug3: hostkeys_foreach: reading file "/Users/mxiong/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/mxiong/.ssh/known_hosts:31
debug3: load_hostkeys: loaded 1 keys from xx.xx.xx.xx
debug1: Host 'xx.xx.xx.xx' is known and matches the RSA host key.
debug1: Found key in /Users/mxiong/.ssh/known_hosts:31
debug2: bits set: 1514/3072
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /Users/mxiong/.ssh/id_rsa (0x7fbf39800000)
debug2: key: /Users/mxiong/.ssh/id_dsa (0x0)
debug2: key: /Users/mxiong/.ssh/id_ecdsa (0x0)
debug2: key: /Users/mxiong/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/mxiong/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /Users/mxiong/.ssh/id_dsa
debug3: no such identity: /Users/mxiong/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /Users/mxiong/.ssh/id_ecdsa
debug3: no such identity: /Users/mxiong/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/mxiong/.ssh/id_ed25519
debug3: no such identity: /Users/mxiong/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I dont find any errors on the ssh log. It seems just stop connection. I try to run

gcloud compute ssh instance1 --ssh-flag="-vvv"

It's basically output same result.

mxiong@myhost ~  ll .ssh
total 72
drwx------    9 mxiong  1742120565    306 Jun 21 15:18 .
drwxr-xr-x+ 109 mxiong  1742120565   3706 Jun 21 15:28 ..
-rw-r--r--    1 mxiong  1742120565     53 Aug 10  2017 config
-rw-------    1 mxiong  1742120565   1679 Jun 21 14:47 google_compute_engine
-rw-r--r--    1 mxiong  1742120565    401 Jun 21 14:47 google_compute_engine.pub
-rw-r--r--    1 mxiong  1742120565    409 Jun 21 14:52 google_compute_known_hosts
-rw-------    1 mxiong  1742120565   1679 Jun 19  2017 id_rsa
-rw-r--r--    1 mxiong  1742120565    414 Jun 19  2017 id_rsa.pub
-rw-r--r--    1 mxiong  1742120565  11739 Jun 19 18:29 known_hosts

Answer 1

You can add your public key into the instance

cat ~/.ssh/google_compute_engine.pub

Then

1. In the Google Cloud Platform Console, go to the VM instances page.
2. Click on the name of the instance that you want to modify.
3. Click Edit in the toolbar.
4. Under SSH Keys, click Show and edit
5. add a public SSH key, click Add item
6. Click Save

Finally, repeat the gcloud ssh command.

gcloud compute ssh instance1 --ssh-flag="-vvv"