How to add a step to emr cluster within a vpc through lambda function which is inside a same vpc and private subnet?

Question

when the lambda function is not within vpc then iam able to add a step to emr cluster but if the lambda function is residing inside vpc where emr cluster is present and same private VPC subnet also.

This time iam getting timeout error when iam trying to add a step in emr cluster using boto3 client module "add_job_flow_steps"

"errorMessage": "2020-05-14T02:48:46.771Z ad979ac2-ff26-476a-b301-23797caeeaa9 Task timed out after 123.10 seconds".

Do i need to add a VPC Endpoint for me to communicate between AWS services within same VPC Subnet or is there any other way which i could communicate?

Answer 1

when the lambda function is not within vpc then iam able to add a step to emr cluster

This works because lambda not in vpc , can access internet. Subsequently, you can connect to public endpoint of AWS services, such as EMR.

if the lambda function is residing inside vpc where emr cluster is present and same private VPC subnet also.

This does not work, because lambda in VPC does not have internet access :

If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address .

To enable your lambda access the EMR service you need to use either NAT gateway or VPC interface endpoint as shown in the following link:

• Connect to Amazon EMR Using an Interface VPC Endpoint

Please not that lambda in VPC also requires modified execution role .