Reading files on EC2 from AWS Lambda

Question

I have some files uploaded on an EC2 instance. I would like to read and process them from a lambda function.

Is there a short snippet (Python preferred) on how to do it? Or just a short tutorial?

I've been through ton of tutorials but I can't figure it out yet

Answer 1

If you decide to use a EC2 instance to serve files, make sure:

• You can reach the instance from outside which means it needs to have a public IP and the server has to listen for connections from anywhere (0.0.0.0)
• EC2 security group has to allow 0.0.0.0 for 80 or 443 (or custom port)

In Python, there is SimpleHTTPServer . You can use it as a starting point and harden it to suit to your needs.

import SimpleHTTPServer
import SocketServer

PORT = 8000

Handler = SimpleHTTPServer.SimpleHTTPRequestHandler

httpd = SocketServer.TCPServer(("", PORT), Handler)

print "serving at port", PORT
httpd.serve_forever()

Answer 2

You would have to serve the files somehow from the EC2 server using a web server or something. Or you could try using an SCP or SFTP library in your Lambda function to connect to the EC2 server and copy the files over. The normal way you would share files between these services is to have a process on the EC2 server copy the files to S3 instead of leaving them on the EC2 server.

Answer 3

You must understand that, as per default, your EC2 volumes are not available to anyone (or anything) outside your instance. They are, for the sake of understand, "local volumes".

Your lambda code runs outside your EC2 instances. You can configure it to run like it is in your VPC, but even in this case it will not have access to your "local" volumes inside your instance.

To give access to the files inside your EC2 to lambda functions you'll have a few options:

• You can install a file sharing service like FTP in yout instance and connnect to it in your lambda;
• You can install a endpoint service to execute tasks on files inside your EC2 and connecto to this service from your lambda.

But the point is: Maybe this design is not the solution for your need, considering all other AWS services available. As told in the previous response, maybe you'll be better served using S3. Or maybe lambda is not the best compute solution. Keep in mind that Lambda has a few time limits to be run and using a disk operation by network will put you in a dangerous zone considering Lambda limitations.

If you can give us more information about your use case we can help you better.