Very strange SSH issue connecting via Ansible to Vagrant VM

Question

I have ansible 2.0.0.2 installed on my Ubuntu 16.04.01 host system.

My hosts file is the following

[testservers]
tsrv1 ansible_ssh_private_key_file=/home/pkaramol/Workspace/vagrans/vagrant/precise32/.vagrant/machines/default/virtualbox/private_key ansible_user=vagrant

[dbservers]
dmsdb_1 ansible_ssh_private_key_file=/home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key ansible_user=vagrant

While ansible tsrv1 -m ping is successful, ansible dmsdb_1 -m ping -vvvv yields the following weird output

(note that I run both after deleting ~/.ssh/known_hosts file)

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016\r
debug1: Reading configuration data /home/pkaramol/.ssh/config\r
debug1: /home/pkaramol/.ssh/config line 1: Applying options for *\r
debug1: Reading configuration data /etc/ssh/ssh_config\r
debug1: /etc/ssh/ssh_config line 19: Applying options for *\r
debug1: auto-mux: Trying existing master\r
debug1: Control socket \"/home/pkaramol/.ansible/cp/ansible-ssh-dmsdb_1-22-vagrant\" does not exist\r
debug2: resolving \"dmsdb_1\" port 22\r
debug2: ssh_connect_direct: needpriv 0\r
debug1: Connecting to dmsdb_1 [192.168.33.11] port 22.\r
debug2: fd 3 setting O_NONBLOCK\r
debug1: fd 3 clearing O_NONBLOCK\r
debug1: Connection established.\r
debug3: timeout: 10000 ms remain after connect\r
debug1: key_load_public: No such file or directory\r
debug1: identity file /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key type -1\r
debug1: key_load_public: No such file or directory\r
debug1: identity file /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key-cert type -1\r
debug1: Enabling compatibility mode for protocol 2.0\r
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\r
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1\r
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000\r
debug2: fd 3 setting O_NONBLOCK\r
debug1: Authenticating to dmsdb_1:22 as 'vagrant'\r
debug3: send packet: type 20\r
debug1: SSH2_MSG_KEXINIT sent\r
debug3: receive packet: type 20\r
debug1: SSH2_MSG_KEXINIT received\r
debug2: local client KEXINIT proposal\r
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c\r
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa\r
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc\r
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc\r
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r
debug2: compression ctos: zlib@openssh.com,zlib,none\r
debug2: compression stoc: zlib@openssh.com,zlib,none\r
debug2: languages ctos: \r
debug2: languages stoc: \r
debug2: first_kex_follows 0 \r
debug2: reserved 0 \r
debug2: peer server KEXINIT proposal\r
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519\r
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r
debug2: compression ctos: none,zlib@openssh.com\r
debug2: compression stoc: none,zlib@openssh.com\r
debug2: languages ctos: \r
debug2: languages stoc: \r
debug2: first_kex_follows 0 \r
debug2: reserved 0 \r
debug1: kex: algorithm: curve25519-sha256@libssh.org\r
debug1: kex: host key algorithm: ecdsa-sha2-nistp256\r
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com\r
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com\r
debug3: send packet: type 30\r
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r
debug3: receive packet: type 31\r
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:RS3yHSobFXsfY/LLfBdMIoWNibW8qvoB/Ff9MBkL9/c\r
Warning: Permanently added 'dmsdb_1,192.168.33.11' (ECDSA) to the list of known hosts.\r
debug3: send packet: type 21\r
debug2: set_newkeys: mode 1\r
debug1: rekey after 134217728 blocks\r
debug1: SSH2_MSG_NEWKEYS sent\r
debug1: expecting SSH2_MSG_NEWKEYS\r
debug3: receive packet: type 21\r
debug2: set_newkeys: mode 0\r
debug1: rekey after 134217728 blocks\r
debug1: SSH2_MSG_NEWKEYS received\r
debug2: key: pkaramol@ISA9813 (0x55751be641f0), agent\r
debug2: key: /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key ((nil)), explicit\r
debug3: send packet: type 5\r
debug3: receive packet: type 6\r
debug2: service_accept: ssh-userauth\r
debug1: SSH2_MSG_SERVICE_ACCEPT received\r
debug3: send packet: type 50\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password\r
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r
debug3: authmethod_lookup gssapi-with-mic\r
debug3: remaining preferred: gssapi-keyex,hostbased,publickey\r
debug3: authmethod_is_enabled gssapi-with-mic\r
debug1: Next authentication method: gssapi-with-mic\r
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available
\r
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available
debug1: Unspecified GSS failure.  Minor code may provide more information

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available
debug2: we did not send a packet, disable method\r
debug3: authmethod_lookup gssapi-keyex\r
debug3: remaining preferred: hostbased,publickey\r
debug3: authmethod_is_enabled gssapi-keyex\r
debug1: Next authentication method: gssapi-keyex\r
debug1: No valid Key exchange context\r
debug2: we did not send a packet, disable method\r
debug3: authmethod_lookup publickey\r
debug3: remaining preferred: ,publickey\r
debug3: authmethod_is_enabled publickey\r
debug1: Next authentication method: publickey\r
debug1: Offering RSA public key: pkaramol@ISA9813\r
debug3: send_pubkey_test\r
debug3: send packet: type 50\r
debug2: we sent a publickey packet, wait for reply\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug1: Trying private key: /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key\r
debug3: sign_and_send_pubkey: RSA SHA256:b2ubdyVwK253Fsewqq5WjEjzEeeKADonx4L9xYPRLfI\r
debug3: send packet: type 50\r
debug2: we sent a publickey packet, wait for reply\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug2: we did not send a packet, disable method\r
debug1: No more authentication methods to try.\r
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r
", 
    "unreachable": true
}

tsrv1 is a precise32 vm dmsdb_1 is a rhel72 vm (to which however I used to be able to connect before some random change occurred that messed things up)

I have also tried vagrand destroy --force && vagrant up for the problematic vm.

update: here is my ~/.ssh/config

Host *
    StrictHostKeyChecking no

the /etc/ssh/ssh_config file

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no

I just use a local configuration file for ansible which just points to the inventory

[defaults]
inventory = ./myfolder

The weird thing is that I CAN connect by typing vagrant ssh to that particular machine!

Any help is highly appreciated I don't know even how to start with this ...

Answer 1

Although this is more of a practical workaround than an actual answer to the above problem, here it goes:

The issue seems to be solved by using a different private network ip address for the specific vm, ie by changing the following line in Vagrantfile from

config.vm.network "private_network", ip: "192.168.33.11"

to

config.vm.network "private_network", ip: "192.168.33.13"

as also the subsequent mapping in /etc/hosts file from

192.168.33.11 dmsdb_1

to

192.168.33.13 dmsdb_1