Is there a way to upload large files to Amazon lambda functions using Amazon s3

Question

I have written a python file which produces specific sentences that I would like to use as part of an Alexa skill. This then produces an array of these sentences. I am now trying to create a lambda function which implements these sentences in an Alexa based game format. However, the python file contains many imports which are not native to lambda so cannot be used.

After much reading I attempted to install the import packages/dependencies using the following code (one library as an example):

pip3 install spacy -t .

I then zip the contents of my folder using the following code, before uploading the zip file into a Amazon s3 bucket:

zip -r ../zipped_dir.zip *

This worked initially but once I began installing many imports, the zip file quickly exceeded 100mb and so I am required to upload the zip file using 'AWS CLI, AWS SDK, or Amazon S3 REST API'. I have tried several methods of doing this, and the zip file successfully uploads but when I attempt to integrate it in my lambda function using the 'Code entry type':'Upload a file from Amazon s3', and providing the correct URL, the function does not allow me to save it. I click save and it attempts to do so, but remains orange. I believe this is because the file is too large. I am sure the upload method is correct because I re-tested the process using a zip file smaller than 100mb and uploaded to the lambda function with the associated URL, and that saved successfully.

In order to upload the large file I have tried the following methods. I have used a combination of flask and boto3 in this example.

import boto3 
from flask import Flask, request

app = Flask(__name__)

@app.route('/')
def index():
    return '''<form method=POST enctype=multipart/form-data                                  `       action="upload">
    <input type=file name=myfile>
    <input type=submit>
    </form'''

@app.route('/upload',methods=['POST'])
def upload():
    s3 = boto3.resource('s3')

s3.Bucket('bucket').put_object(Key='file.zip',Body=request.files ['myfile'])

` return '

File saved to S3

'

 if __name__ == '__main__': app.run(debug=True) 

This method also uploads the file successfully into the bucket, but I am unable to save it with the lambda function and URL.

I have also tried to do it from the terminal with this command, which also uploads into the bucket successfully but cannot be saved on the lambda function:

 aws s3 cp /foldername s3://bucketname/ --recursive --include ` "myzip.zip" 

I am sure manually installing all the files for the associated imports is not the most optimal method and so any suggested other methods would be helpful. If there is also a way to run the python file elsewhere, and pass the array of strings into the lambda function which can run on an Alexa device, that method will also be helpful. I have been stuck on this for almost a week and i'm sure the solution is rather simple, so any help is much appreciated. Thank you.

Answer 1

You're hitting the Lambda Limits with you Archive size > 50MB, that's why your current attempts were unsuccessful.

From the docs :

Deployment package size

50 MB (zipped, for direct upload)

250 MB (unzipped, including layers)

3 MB (console editor)

If you have larger dependencies I suggest you look into using Lambda Layers they basically provide a way to separate your dependencies from your main code.

To make your life easier I recommend you look into using the open source Serverless Framework which makes deploying Lambda functions quite easy. I use the Serverless Framework in combination with the serverless-python-requirements plugin to separate my code from my requirements and deploy the requirements as a Lambda Layer.

Note : Make sure that your unzipped requirements and code stay below 250MB, otherwise you'll hit another limit.