AWS Network Load Balancer failed to connect with EC2 instance if EC2 instance is not open to public

Question

I have been struggling with this problem for 2 days but couldn't get it working.

I have this flow:

external world --> AWS API Gateway ---> VPC Link ---> Network Load Balancer ---> my single EC2 instance

Before introducing the API Gateway, I want to first make sure the Network Load Balancer --> my single EC2 instance part works.

I have set up the EC2 instance correctly. There is a Typescript / ExpressJS api service running on port 3001

I have also set up a Network Load Balancer and a Target Group, the NLB is listening and forwarding requests to port 3001 of the target group (which contains the EC2 instance).

Here is the NLB:

Note that the NLB has a VPC. This raise the question below and I find it so confusing.

listener:

You can see it is forwarding requests to docloud-backend-service , which is described as follows:

You can see that the health check has passed.

I have configured the security group of my EC2 instance with this rule:

1. Allow All protocol traffic on All ports from my VPC 
(specified using CIDR notation `171.23.0.0/16`);

Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user , the command fails by timeout.

Then, after I add this rule:

2. Allow All protocol traffic on All ports from ANY source (`0.0.0.0/0`);

Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user ,

the api service gets the request and I can see logs generated in the EC2 instance.

Question:

The second rule opens up the EC2 instance to public, which is dangerous.

I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.

The NLB has no security group to be configured. It has a VPC though. If I limit the EC2 instance such that only its own VPC can access it, it should be fine, right?

The first rule does exactly that. Why does it fail?

The NLB has a VPC. Requests go from API Gateway to NLB, then from NLB to EC2 instance. So from the EC2 instance's perspective, the requests come from an entity in the VPC. So the first rule should work, right?

Otherwise why would AWS assign a VPC to the NLB anyways?

Why would I see the VPC on the NLB's description console anyways?

Answer 1

I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.

For instance based target groups and for IP based target groups as well we can enable/disable if want to preserve the requester's IP address:

This setting can be found if go to our target group -> Actions -> Edit Target attributes.

What does this mean from the perspective of the Security Group of our application?

If we enable it (which is the default for instance type target groups), the application will see traffic as it is coming directly from the end-client. This means, you we have to enable inbound traffic for 0.0.0.0:3001 .

If we disable it, the application will see the source traffic as it was coming from the private IP address of the Network Load Balancer. In this case, we can limit the inbound traffic to the private IP address of the NLB or to the CIDR range of the subnet in which the NLB is placed.