在S3触发的AWS中运行ETL python脚本

Question

I am new with AWS and don't know how to do the following. When I put an object in S3 I want to launch a python script that does some transformations and returns it to another path in S3. I've tried a lambda function but the process takes more than 300 seconds. I've also tried it with a Glue job but I don't know how to trigger it when I put the file in S3.

Does anyone know how to do it? Maybe I'm using the wrong AWS tools.

Answer 1

One option would be to use SQS :

1. Create the SQS queue.
2. Setup S3 to send notifications to the SQS queue when new objects are added to the source bucket. See Configuring Amazon S3 Event Notifications .
3. Setup your Python script on an EC2 instance and listen to the SQS queue in your code.
4. Upload the output of your Python script into the target S3 bucket after script finished.

Can you break up the Python processing into smaller steps? I'd definitely recommend that you use Lambda instead of managing EC2 if you can get your code to run within the Lambda restrictions.

Answer 2

The simple solution for your problem is here: Since you've already mentioned that you have AWS Glue job working to do this operation. And all you don't know is how to trigger glue job when file placed in s3, I am answering to that question. You can write an AWS lambda using boto3 module which can be triggered based up on the s3 event and have setup glue.start_job_run command in your lambda function.

response = client.start_job_run(
    JobName='string')

https://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.start_job_run

Note:: I strongly believe Glue is the right tool rather than lambda for your requirement that you mentioned in question, because AWS lambda have time out limitation. It will get timeout after 300 seconds.