加载 AWS Glue S3 源数据

Question

I have a use case where AWS Glue is a good fit for data transformation.

However, the source file for this transformation job is retrieved via a HTTPs call which can take 45 mins to return.

What is the best approach to load this data to S3 and then sftp the glue output once completed?

This job needs to be both scheduled and run on demand.

Answer 1

I don't think there's a way of loading the data directly from Glue for HTTP/s now.

You can create a lambda or a EC2 instance with a service that extracts that source file and puts that file into an s3 bucket.

I would suggest you to do this:

• Create an empty S3 bucket
• Create a lambda or an EC2 instance with a service that extracts the source file and loads it into the previous S3 bucket
• Configure the previous bucket with notifications when a new file is PUT/uploaded into the bucket so it notifies an SNS topic.
• Create an AWS lambda subscribed to the previous SNS topic that triggers the Glue Job with your transformations.
• Create a Glue Job that loads the data from the S3 bucket you create, do the transformations and load it into S3.