与 S3 的 AWS Glue 连接失败

Question

I want to create a Glue job to transfer data from RDS PostgreSQL into S3.
To do this, I did the following:

1. Created a Glue connection to RDS PostgreSQL instance
2. Set up a VPC S3 endpoint and NAT gateway (because test connection to RDS was failing to to VPC S3 endpoint and NAT gateway not being present)
3. Created a Glue "Network" connection to S3

When I try to test the connection, I get the following:

Couple of things to note:
For the sake of making it work at all, the role used for testing connection has FullAccess policies to Glue and to S3.
The security group used in the connection is the same one from the RDS PostgreSQL connection (ie allows the TCP connections on port 5432 from my IP, all TCP from itself and all TCP from all IPv4).
The logs in the screenshot, when the cloudwatch is opened, are throwing an AWS error saying that the logs in question don`t exist.
When trying to run a job, without using the S3 connection, the job ends with an error - " An error occurred while calling o96.pyWriteDynamicFrame. connect timed out ". This error is thrown from Java code by a method used to send an HTTP request to the bucket specified in the job, because, apparently, spark is unable to reach the bucket via HTTP.

PS
I`m very new to AWS, having only a little experience with Azure before

Answer 1

It's possible that the security group configuration is not correct for the Glue S3 connection. The security group used for the S3 connection requires ingress/egress configuration for inbound/outbound traffic. A simple way to configure this is to allow ingress/egress for all protocols.

Example ingress configuration:

• from port: 0
• to port: 0
• protocol: -1 (all protocols)
• self: true

Example egress configuration:

• from port: 0
• to port: 0
• protocol: -1 (all protocols)
• CIDR IP: "0.0.0.0/0"

I recommend creating a separate security group for the Glue S3 connection and not reusing the one for RDS.