AWS Lambda - use Kinesis under VPC

Question

I have an AWS Lambda function that makes use of an ElastiCache Redis cluster. Since the Redis cluster is "locked" in a VPC , the Lambda function must reside in that VPC too.

For some reason, if the Lambda is allocated an IP of a public subnet , which has an Internet gateway - it still cannot make connections to the outside (the internet), thus making it impossible to use Kinesis .

For that, they suggest using a NAT gateway which lets the Lambda connect to the outside.

Basically, this works for me - but my issue is the money. This solution is expensive for large amount of data transfers and I'm looking for some way to make it cheaper.

For a small POC that I've made, I paid ~$10 .

This is too much for ~30GB as my production pipeline will run hundreds of gigabytes / month.

How do you suggest I let the Lambda function connect the outside (specifically Kinesis ) without using a NAT gateway?

Thank you!

Answer 1

without using a NAT gateway?

Use a NAT instance .

You have to have one of these two things for anything in VPC to access the Internet from a private IP address.

NAT instances were exactly how this was always done in VPC, until the relatively new NAT Gateway service was rolled out.

You can also use a NAT gateway, which is a managed NAT service that provides better availability, higher bandwidth, and requires less administrative effort. For common use cases, we recommend that you use a NAT gateway rather than a NAT instance.

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html

Sure, it's easier, but it costs more. A lot more. The most significant difference in this case is that with a NAT instance, you pay a flat rate for use of the hardware, which could be an inexpensive t2.nano, $5/mo.

The NAT Gateway service is a high powered solution with nearly infinite scaling capacity, and is priced accordingly. A NAT instance is only as good as the hardware you choose to run it on, but I find t2.nano and t2.micro quite adequate for workloads requiring less than 250 Mbit/s of Internet connectivity.

Use the link, above, to learn more.

Answer 2

Lambda function instances will never be assigned a public IP address, regardless of the type of VPC subnet you place them in. A NAT gateway is the only solution to provide a Lambda function inside a VPC with access to resources that reside outside the VPC (like Kinesis).

If that isn't going to work for you due to cost, you might look into running a Redis server on an EC2 instance with an Elastic IP, which would allow the Lambda function to connect without being inside the VPC. A similar alternative would be to use RedisLabs instead of ElastiCache.