从 S3 存储桶导入 AWS Lambda 函数代码中的库

Question

I have to deploy a Deep Learning model on AWS Lambda which does object detection. It is triggered on addition of image in the S3 bucket. The issue that I'm facing is that the Lambda function code uses a lot of libraries like Tensorflow, PIL, Numpy, Matplotlib, etc. and if I try adding all of them in the function code or as layers, it exceeds the 250 MB size limit. Is there any way I can deploy the libraries zip file on S3 bucket and use them from there in the function code (written in Python 3.6) instead of directly having them as a part of the code? I can also try some entirely different approach for this.

Answer 1

It sounds to me like Amazon SageMaker would be a better choice for your task.

You can create a model and host it on an endpoint all via SageMaker. The use a lambda function triggered by your s3 upload to pass the image to your SageMaker endpoint and process the result.

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning (ML) models quickly. SageMaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high quality models.

https://aws.amazon.com/sagemaker/

Answer 2

Try out pre-compiled packs for Tensorflow, they are about 50MB, you probably would have enough space for the rest of modules you need. And in general, check the AWS recommendations on deep learning architecture approaches.

Answer 3

Ultimately, I changed my approach from using Lambda to using EC2. I deployed the whole code with libraries on an EC2 instance and then triggered it using Lambda. On EC2, it can also be deployed on Apache server to change port mapping.