How to Store Large Python Dependencies on S3 (for AWS Lambda with Serverless)

Question

I am using AWS Lambda to host a Python project, managing deployments using the Serverless framework, and have come up against the commonly-hit 50MB package storage limits. Until now, I've successfully split my requirements up per function using the serverless-python-individually and serverless-python-requirements plug-ins to dockerise, compress and upload each file as a separate Lambda function, as shown in the key section of my serverless.yml file, shown below.

This has kept the first three function packages below 50MB, and created a simple workflow wherein I can push changes simply using: sls deploy --pi-dockerizedPip . However, the fourth package, with only the requirement for SciPy , cannot be reduced below 52MB, and after much reading the only common approach I see is to upload my dependency(ies?) to S3 (somewhere) and somehow ensure my import statements are able to find the dependencies.

I haven't found clear instructions on exactly how this process of packaging dependencies should be done, and (almost as important), how can will work alongside Serverless (so I can keep the nice workflow)?

Key sections of serverless.yml:

package:
  individually: True
  exclude:
    # Exclude everything first.
    - '**/*'

functions:
  brain:
    handler: src/1-brain/wrap.handler
    package:
      include:
        - src/1-brain/**
    events:
      - schedule: rate(5 minutes)
  data:
    handler: src/2-data/wrap.handler
    package:
      include:
        - src/2-data/**
  strategy:
      handler: src/3-strategy/wrap.handler
      package:
        include:
          - src/3-strategy/**
  balancer:
      handler: src/4-portfolio-balancer/wrap.handler
      package:
        include:
          - src/4-portfolio-balancer/**  
custom:
  pythonRequirements:
    useDownloadCache: true
    useStaticCache: true
    dockerizePip: true
    zip: true
    slim: true
  pyIndividually:
    wrap:brain: src/1-brain/aws_handler.handler     # mapping to the real handler
    wrap:data: src/2-data/datafeed.handler          # mapping to the real handler
    wrap:strategy: src/3-strategy/strategy.handler  # mapping to the real handler
    wrap:balancer: src/4-portfolio-balancer/balancer.handler  # mapping to the real handler

Answer 1

AWS Lambda Layers enable you to bundle your packages/environment into a layer. You can use up to 5 layers, and have a 250 MB uncompressed limit across the cumulative sum of all respective layers + lambda code for any particular lambda function.

Check out the section on "Including Library Dependencies in a Layer" https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html