从 Python 3.8 中的 AWS lambda 临时目录导入 package

Question

The problem I have is similar to this SO question but the answer doesn't work for me. I am trying to import Python library (let's say xgboost ) from /tmp folder in AWS Lambda.

Library requests is added to Lambda layer and what I did is:

import json
import io
import os
import zipfile
import requests
import sys

sys.path.insert(0, '/tmp/')
sys.path.append('/tmp/')

os.environ["PYTHONPATH"] = "/var/task"

def get_pkgs(url):
    print("Getting Packages...")
    re = requests.get(url)
    z = zipfile.ZipFile(io.BytesIO(re.content))
    print("Extracting Packages...")
    z.extractall("/tmp/")
    print("Packages are downloaded and extracted.")
    
def attempt_import():
    print("="*50)
    print("ATTEMPT TO IMPORT DEPENDENCIES...")
    print("="*50)
    import xgboost
    print("IMPORTING DONE.")
    
def main():
    URL = "https://MY_BUCKET.s3.MY_REGION.amazonaws.com/MY_FOLDER/xgboost/xgboost.zip"

    get_pkgs(URL)
    attempt_import()
    
def lambda_handler(event, context):
    main()
    return "Hello Lambda"

The error I get is [ERROR] ModuleNotFoundError: No module named 'xgboost' . I gave my S3 bucket all necessary permissions, and I am positive that Lambda can access the .zip file since the requests.get works and variable z returns:

<zipfile.ZipFile file=<_io.BytesIO object at 0x7fddaf31c400> mode='r'>

Answer 1

You could try using the boto3 library to download the file from S3 to /tmp directory as explained in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.download_file

import boto3
s3 = boto3.resource('s3')
s3.meta.client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

Answer 2

Actually, my code above works and I had a rather silly error. Instead of zipping the xgboost package folders ( xgboost , xgboost.libs and xgboost.dist-info ) I actually zipped their parental folder which I named package-xgoboost , and that didn't worked in AWS lambda.Be sure that you actually zip those 3 folders directly.

Also, make sure your xgboost library is up-to-date. Previously I used version 1.2.1 which didn't work either. Upgrading the library and zipping the newest xgboost version (in my case 1.3.3) finally worked.