SSH Tunnel through Ubuntu bastion to instance in private subnet

Question

I stocked in problem which I can't fully get how to solve. The question is very common, but I still can't get what am I doing wrong.

According this AWS doc: Scenario 2: VPC with Public and Private Subnets (NAT) I have my own VPC with two subnets: private and public. In public subnet I have deployed an Ubuntu 16.04 Instance with assigned EIP. It also has next security group inbound rules:

Type   Protocol Port Range Source            Description
SSH    TCP      22         xx.xx.xx.xx/32    Home IP

and outbound accordingly:

Type   Protocol Port Range Source            Description
SSH    TCP      22         sg-xxprivatexx    Security group ID for instance in private subnet

Looks nice, I can ssh it externally from my home. No problem.

In private subnet I have deployed another one Ubuntu 16.04 machine with next security group (inbound rules):

Type   Protocol Port Range Source            Description
HTTP   TCP      80         sg-xxpublicxxx    Security Group ID for bastion instance in public subnet
SSH    TCP      22         sg-xxpublicxxx    -

and no outbound rules (actually it has 80, 443 outbound ports opened, but its not an interesting part as I guess). And I still can reach this virtual machine using ssh from my bastion.

Right now I just want to make only one simple thing - run ssh port forwarding so I can run localhost:8080 on my home PC browser and see the webpage I published on my private instance. If I understand it correctly from here and here (and from here as well) I have to run something like:

 ssh -N -v -L 8080:10.0.1.112:80 ubuntu@3.121.46.99

Which as I guess basically means: just forward a traffic from private subnet instance with IP 10.0.1.112:80 to my localhost:8080 through my bastion VM with username ubuntu hosted on EIP 3.121.46.99 .

Debug ends with lines:

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:ZyVHgnF8z5vE5gfNr1S2JDfjhdydZVTNevPRgJZ+sRA /home/matterai/.ssh/key.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/matterai/.ssh/id_rsa
debug1: Trying private key: /home/matterai/.ssh/id_dsa
debug1: Trying private key: /home/matterai/.ssh/id_ecdsa
debug1: Trying private key: /home/matterai/.ssh/id_ed25519
debug1: No more authentication methods to try.
matterai@3.121.46.99: Permission denied (publickey).

I am playing around it few days and I still can't get what am I doing wrong. Its so strange: I can ssh -A (to allow forwarding) to my bastion, I can ssh to my private instance from bastion. But I cant establish SSH tunnel to see my webpage (in the future it will be mongodb) without an error. Need some advice or point to the right direction, please! Thank you.

UPD#1

Ok then. If I make manual forwarding using my local machine and my bastion, I get an expected result. Basically it means run this command on bastion:

ubuntu@bastion: ssh -v -N -L 5000:localhost:8000 ubuntu@10.0.1.68

After that runs command on local/home machine:

matterai@homepc: ssh -v -N -L 5000:localhost:5000 ubuntu@3.121.46.99

When I make a request to localhost:5000 on my local machine, I can see the result page. May I and how if it's possible to combine this two commands? (spoiler: yes, it's possible: see the answer!)

Answer 1

Ok, it's easy. Hope my answer will help somebody.

1. You need to use ssh -J option to connect through your bastion virtual machine:

  -J [user@]host[:port] Connect to the target host by first making a ssh connection to the jump host and then establishing a TCP forwarding to the ulti‐ mate destination from there. Multiple jump hops may be specified separated by comma characters. This is a shortcut to specify a ProxyJump configuration directive. 

2. Then you need to forward traffic from your destination virtual machine port ( :8000 ) where the app (or database) started to your localhost port ( :5001 ) using ssh -L :

  -L [bind_address:]port:host:hostport -L [bind_address:]port:remote_socket -L local_socket:host:hostport -L local_socket:remote_socket Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side. This works by allocating a socket to listen to either a TCP port on the local side, optionally bound to the specified bind_address, or to a Unix socket. Whenever a connection is made to the local port or socket, the connection is forwarded over the secure channel, and a connection is made to either host port hostport, or the Unix socket remote_socket, from the remote machine. Port forwardings can also be specified in the configuration file. Only the superuser can forward privileged ports. IPv6 addresses can be specified by enclosing the address in square brackets. By default, the local port is bound in accordance with the GatewayPorts setting. However, an explicit bind_address may be used to bind the connection to a specific address. The bind_address of “localhost” indicates that the listening port be bound for local use only, while an empty address or '*' indicates that the port should be available from all interfaces. 

3. Full ssh command will look like:

matterai@homepc: ssh -v -N -A -J ubuntu@3.121.46.99 -L 5001:localhost:8000 ubuntu@10.0.1.112

Answer 2

You seem to have things correctly configured, but the error is saying that it can't find a private key to use for the connection.

To test the port forwarding, start by using the ssh command that logs into your public instance.

Then, take that exact command , and simply add: -L 8080:10.0.1.112:80

If it works for 'normal' ssh, then it will work with port forwarding too.

By the way, in general you should never need to modify the outbound rules of a security group. The default settings permit all outbound traffic . This 'trusts' the apps running on the instance and allows them to communicate outwards to anywhere. You would only need to restrict such rules where you wish to enforce a high-security environment.