如何触发Lambda只有一次多部分上传S3触发器？

Question

I have a AWS Lambda function that:

1. reacts to an S3 ObjectCreated trigger
2. ssh into an ec2 instance and
3. runs a python script

This python script will then run EMR to process all these S3 part-files that were just created. (Yes, the files must be processed jointly.)

However, the part-files ( file_part_0000 , file_part_0001 , file_part_0002 , etc.) are created as multipart uploads.

Lambda Event: {..., "eventName": "ObjectCreated:CompleteMultipartUpload", ...}

It's my understanding that this trigger would force Lambda to ssh and run the python script as many time as there are part files. I only want Lambda to ssh and run the script once .

Is there any way to have the Lambda be triggered once for each "series" of multipart uploads?

Answer 1

As mentioned in Configuring Amazon S3 Event Notifications

You will not receive event notifications from failed operations.

And there is a reason that the event is call s3:ObjectCreated:CompleteMultipartUpload , the event should trigger only if your multipart are upload completely.

On the other hand, perhaps you should make S3 event trigger to send info to SQS. So you can make your python program can watching SQS queue and process the file. With SQS, you don't need to make passwordless setup or store private key in lambda to ssh the server.