How can I access aws resources in VPC from AWS glue?

Question

I have a glue job which is hitting an API hosted over an EC2 instance.

The problem is EC2 instance resides within a VPC blocking all public access.

I tried creating an endpoint interface in my VPC but still can't access the REST API.

The host is always unreachable but when I try to access the API from VPC it is working fine.

The security group associated with the EC2 instance is used while creating the VPC Endpoint.

Any help is appreciated

Answer 1

If you go to AWS Glue console, under connections, create a connection. What is meant by a dummy connection, is just be a non-existent database or resource for example: jdbc:mysql://some-fake-endpoint-here:3306/mydb . After this you choose the correct VPC, subnet and security group. Which means a test connection will not work in this context but what it brings is a way to introduce your VPC, Subnet and Security group information to the job. Testing such a connection can be done using a python-shell job or launch an ec2 instance in the same vpc or same subnet and run something like nc -vz endport port .

This connection metadata information will facilitate the launching of elastic network interfaces in your account that allow glue DPUs to communicate with your resource at runtime. More on how connections in glue is discussed here .