How to copy a file from S3 bucket to an EC2 Windows instance using AWS Lambda function

Question

I am trying to copy the file(s) uploaded to S3 to the EC2 windows instance via AWS Lambda function. My requirement is the AWS Lambda function should trigger when a file has been uploaded to a S3 bucket and the same file should be copied to the EC2 instance.

How the AWS Lambda function can be connected to an EC2 Windows instance ? Please let me how can I achieve this scenario.

Answer 1

You can approach this problem in two ways:

1. Creating a Shared File System for your Lambdas and EC2. https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/

2. Creating an API service on your EC2 to receive Files.

The first approach is the most secure and probably the better way of doing it. The second approach is funnier. You could create a Node+Express API that receives files and just sending it from your Lambda in a POST request.

Answer 2

An AWS Lambda function runs outside of EC2 instances. In fact, you can think of it as a very temporary EC2 instance.

Therefore, the answer to your question is "Well, how would you copy something from an EC2 Linux instance to a Windows instance?"

The biggest issue is that good security means that it's not easy to "push" things into an EC2 instance. For example, I can't push things into the computer you are currently using. Therefore, the best answer is that instead of "pushing" the file, it is better to "pull" the file. Here's a few ways:

Systems Manager Run Command

The AWS Lambda function could trigger the AWS Systems Manager Run Command , which can execute a script on the EC2 instance. This script could 'pull' the file from S3 onto the instance.

This is made possible because the EC2 instance has the Systems Manager Agent installed, which can receive the request and trigger the script.

Polling an Amazon SQS queue

The AWS Lambda function could send a message to an Amazon SQS queue. A script on the instance could be continually polling the SQS queue looking for a message. When it receives a message, it uses the information to 'pull' the file from S3 into the instance. (Use Long Polling on the SQS queue to reduce the number of requests.)

In fact, Amazon S3 can send the message directly to the Amazon SQS queue without requiring the Lambda function.

Use a Web Server

The AWS Lambda function could POST the file via a web server running on the EC2 instance that is 'listening' for such requests.

Shared filesystem

As suggested by @mimetist, an AWS Lambda function can be connected to an Amazon EFS filesystem. Thus, it could store the file on the filesystem, which could instantly make it accessible to the EC2 instance if it also has the same filesystem mounted.

Answer 3

The issue is fixed now by using the AWS Lambda with S3 and Systems manager worked for me. I have created a node.js script to get the auto event from s3 and push back the details to a systems manager run powershell script document and thereby copying to ec2 instance which was worked as expected.

You need to chose the right instance type based on the file transfer from s3 to EC2