AWS Glue ETL job from AWS Redshift to S3 fails

Question

I am trying out AWS Glue service to ETL some data from redshift to S3. Crawler runs successfully and creates the meta table in data catalog, however when I run the ETL job ( generated by AWS ) it fails after around 20 minutes saying "Resource unavailable".

I cannot see AWS glue logs or error logs created in Cloudwatch. When I try to view them it says "Log stream not found. The log stream jr_xxxxxxxxxx could not be found. Check if it was correctly created and retry."

I would appreciate it if you could provide any guidance to resolve this issue.

Answer 1

So basically, the job you add to Glue will either run if there's not too much traffic in the region your Glue is. If there are no resources available, you need to either manually re-add the job again or you can also bind yourself to events from CloudWatch via SNS .

Also, there are parameters you can pass to the job like maximunRetry and timeout .

If you have a Ressource not available , it won't trigger a retry because the job did not fail, it just didn't even started. But if you set the timeout to let's say 60 minutes , it will trigger an error after that time, decrement your retry pool and re-launch the job.

Answer 2

The closest thing I see to Glue documentation on this is here:

If you encounter errors in AWS Glue, use the following solutions to help you find the source of the problems and fix them. Note The AWS Glue GitHub repository contains additional troubleshooting guidance in AWS Glue Frequently Asked Questions. Error: Resource Unavailable If AWS Glue returns a resource unavailable message, you can view error messages or logs to help you learn more about the issue. The following tasks describe general methods for troubleshooting. • A custom DNS configuration without reverse lookup can cause AWS Glue to fail. Check your DNS configuration. If you are using Amazon Route 53 or Microsoft Active Directory, make sure that there are forward and reverse lookups. For more information, see Setting Up DNS in Your VPC (p. 23). • For any connections and development endpoints that you use, check that your cluster has not run out of elastic network interfaces.

Answer 3

I have recently struggled with Resource Unavailable thrown by Glue Job

Also i was not able to make a direct connection in Glue using RDS -it said "no suitable security group found"

I faced this issue while trying to connect with AWS RDS and Redshift

The problem was with the Security Group that the Redshift was using. There is a need to place a self referencing inbound rule in the Security Group.

For those who dont know what is self referencing inbound rule, follow the steps

1) Go to the Security Group you are using (VPC -> Security Group)

2) In the Inbound Rules select Edit Inbound Rules

3) Add a Rule

a) Type - All Traffic b) Protocol - All c) Port Range - ALL d) Source - custom and in space available write the initial of your security group and select it. e) Save it.

Its done !

if you were missing this condition in your Security Group Inbound Rules

Try creating the connection you will be able to create the connection.

Also job should work this time.