如何在特定时间间隔从 S3 存储桶检查和下载文件？

Question

I want to implement the following once a file is uploaded to aS3 Bucket

1. Download the file to a windows server
2. Run a 3rd party exe to process the file and generate an output file on a Windows Server

What is the best approach to implement this using.Net Core?

Solution 1:

1. Create a Lambda function to Trigger an API
2. API will download the file and process

Solution 2:

1. Create an executable to download the file from s3 bucket
2. Create a lambda function trigger an executable

Solution 3:

1. Create a service to check and download files from s3 bucket
2. The downloaded file will be processed by the service

Solution 4:

1. Use AWS Lambda to push the file to SQS
2. Create an application to monitor SQS.

Please let me know the best solution to implement this. Sorry for asking this non-technical question.

Answer 1

The correct architecture approach would be:

• Create a trigger on the Amazon S3 bucket that sends a message to an Amazon SQS queue when the object is created
• A Windows server is continually polling the Amazon SQS queue waiting for a message to appear
• When a message appears, use the information in the message to download the object from S3 and process the file
• Upload the result to Amazon S3 and optionally send an SQS message to signal completion (depending on what you wish to do after a file is processed)

This architecture is capable of scaling to large volumes and allows files to be processed in parallel and even across multiple servers. If a processing task fails and does not signal completion, then Amazon SQS will make the message visible again for processing.