在无服务器部署时自动安装依赖项

Question

I'm using Serverless v2.23.0 to deploy a bunch of endpoints to AWS Lambda functions. The Lambda functions run python 3.8. I want to automatically install a dependency upon deploy to AWS Lambda.

For example, I want the package pycurl installed automatically. I am following the tutorial at https://www.serverless.com/blog/serverless-python-packaging

I added a requirements.txt file which just has the line:

pycurl

To serverless.yml I added:

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

When I deploy via serverless, I get this monstrosity:

STDERR:     ERROR: Command errored out with exit status 1:
     command: /var/lang/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-jv_aqotm
         cwd: /tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/
    Complete output (22 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py", line 236, in configure_unix
        p = subprocess.Popen((self.curl_config(), '--version'),
      File "/var/lang/lib/python3.8/subprocess.py", line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/var/lang/lib/python3.8/subprocess.py", line 1702, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py", line 988, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py", line 649, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py", line 101, in __init__
        self.configure()
      File "/tmp/pip-install-_p_d5byz/pycurl_960a4c2e1a5b4bf1b382d3930bd2fba3/setup.py", line 241, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/50/1a/35b1d8b8e4e23a234f1b17a8a40299fd550940b16866c9a1f2d47a04b969/pycurl-7.43.0.6.tar.gz#sha256=8301518689daefa53726b59ded6b48f33751c383cf987b0ccfbbc4ed40281325 (from https://pypi.org/simple/pycurl/) (requires-python:>=3.5). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /var/lang/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-6g6vfjsr
         cwd: /tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/
    Complete output (22 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py", line 233, in configure_unix
        p = subprocess.Popen((self.curl_config(), '--version'),
      File "/var/lang/lib/python3.8/subprocess.py", line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/var/lang/lib/python3.8/subprocess.py", line 1702, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py", line 961, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py", line 623, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py", line 101, in __init__
        self.configure()
      File "/tmp/pip-install-_p_d5byz/pycurl_fe88f280458f4a83807016624ffa0f80/setup.py", line 238, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config'
    ----------------------------------------
  ...
  ...
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/08/79/576eaecd19f11a265cd8d10f339ecfd4b1033436e56e6f2625153a4b79aa/pycurl-7.19.0.2.tar.gz#sha256=7a9e793b9181654d5eef3f6d22c244c57d2b51d38feb4c1b71d68efda99b0547 (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /var/lang/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_p_d5byz/pycurl_94d64ac946c44540bb8e097906871c69/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_p_d5byz/pycurl_94d64ac946c44540bb8e097906871c69/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1gyrq4uv
         cwd: /tmp/pip-install-_p_d5byz/pycurl_94d64ac946c44540bb8e097906871c69/
    Complete output (6 lines):
    /bin/sh: curl-config: command not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_p_d5byz/pycurl_94d64ac946c44540bb8e097906871c69/setup.py", line 90, in <module>
        raise Exception("`%s' not found -- please install the libcurl development files or specify --curl-config=/path/to/curl-config" % CURL_CONFIG)
    Exception: `curl-config' not found -- please install the libcurl development files or specify --curl-config=/path/to/curl-config
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/17/11/79e2187232ffdb96c9c200f8fc82452f8631f659fd9bdd2acf4bf7991905/pycurl-7.19.0.1.tar.gz#sha256=8b14d91d7453c5d9a4cc8cd4e61ab9389298813028d10661bf2079ee06d676c8 (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /var/lang/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_p_d5byz/pycurl_a3d77a6dfa274613baa311998e8830a2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_p_d5byz/pycurl_a3d77a6dfa274613baa311998e8830a2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-skxkhoag
         cwd: /tmp/pip-install-_p_d5byz/pycurl_a3d77a6dfa274613baa311998e8830a2/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_p_d5byz/pycurl_a3d77a6dfa274613baa311998e8830a2/setup.py", line 58
        print "FATAL: bad directory %s in environment variable %s" % (dir, envvar)
              ^
    SyntaxError: invalid syntax
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/11/73/abcfbbb6e1dd7087fa53042c301c056c11264e8a737a4688f834162d731e/pycurl-7.19.0.tar.gz#sha256=8ef0bf1c0e48f0d193ef8ac4ae4b3596a1e91b6cbf1ff1b25de769fe3782aeb6 (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /var/lang/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_p_d5byz/pycurl_c9a9d43d5c9a4fb4b41908f9f1558869/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_p_d5byz/pycurl_c9a9d43d5c9a4fb4b41908f9f1558869/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-mnx6l7bp
         cwd: /tmp/pip-install-_p_d5byz/pycurl_c9a9d43d5c9a4fb4b41908f9f1558869/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_p_d5byz/pycurl_c9a9d43d5c9a4fb4b41908f9f1558869/setup.py", line 58
        print "FATAL: bad directory %s in environment variable %s" % (dir, envvar)
              ^
    SyntaxError: invalid syntax
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/fe/8f/711d436c76938093dd10dae465ccee316c6241cfdada06c5cc3b5c1f74b8/pycurl-7.18.2.tar.gz#sha256=cda1085b7911ca802746d816f37d541165ad1e20c241d22e4c304ddb2bc81ffe (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pycurl
ERROR: No matching distribution found for pycurl

    at /Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/serverless-python-requirements/lib/pip.js:325:13
    at Array.forEach (<anonymous>)
    at installRequirements (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/serverless-python-requirements/lib/pip.js:312:28)
    at installRequirementsIfNeeded (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/serverless-python-requirements/lib/pip.js:556:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/serverless-python-requirements/lib/pip.js:635:29)
    at ServerlessPythonRequirements.tryCatcher (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/markkelly/RedMatter/repos/new_version/RedMatterAppBackend/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:439:21)
    at process.topLevelDomainCallback (domain.js:130:23)

   For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Get Support --------------------------------------------
   Docs:          docs.serverless.com
   Bugs:          github.com/serverless/serverless/issues
   Issues:        forum.serverless.com

Your Environment Information ---------------------------
   Operating System:          darwin
   Node Version:              12.14.1
   Framework Version:         2.23.0
   Plugin Version:            4.4.3
   SDK Version:               2.3.2
   Components Version:        3.6.2

Picking from the debris, I see errors such as:

Exception: `curl-config' not found -- please install the libcurl development files or specify --curl-config=/path/to/curl-config

and:

ERROR: Could not find a version that satisfies the requirement pycurl
ERROR: No matching distribution found for pycurl

I tried other dependencies using this method, and had no issues at all. Why is pycurl not being installed?

EDIT

I implemented the answer here and it deployed correctly. But when I do a CURL to the endpoint, I get:

[ERROR] Runtime.ImportModuleError: Unable to import module 'handler': No module named 'pycurl'

Handler is:

import os
import json
import uuid
import time
import boto3
from boto3.dynamodb.conditions import Key
import pycurl
from io import BytesIO 



def events(event, context):

    b_obj = BytesIO() 
    crl = pycurl.Curl() 

    # Set URL value
    crl.setopt(crl.URL, '[url to fetch]')

    # Write bytes that are utf-8 encoded
    crl.setopt(crl.WRITEDATA, b_obj)

    # Perform a file transfer 
    crl.perform() 

    # End curl session
    crl.close()

    # Get the content stored in the BytesIO object (in byte characters) 
    get_body = b_obj.getvalue()

    print(get_body)

    response = {
        "statusCode": "200",
        "headers": {
            "Access-Control-Allow-Origin": "*",
            "Content-Type": "application/json"
            },
        "body": json.dumps(get_body)
    }

    return response

serverless.yml:

service: rmappbackend
frameworkVersion: "2"

package:
exclude:
    - "frontend/**"

plugins:
- serverless-python-requirements

custom:
pythonRequirements:
    dockerizePip: true
    dockerFile: ./dockerfile

provider:
name: aws
runtime: python3.8
stage: ${opt:stage}
lambdaHashingVersion: 20201221
apiGateway:
    shouldStartNameWithService: true
environment:
    STAGE: ${opt:stage}
    EVENTS_TABLE: ${opt:stage}-events
    GRAPH_TABLE: ${opt:stage}-graph
    DATA_TABLE: ${opt:stage}-data
    STORAGE_BUCKET: ${opt:stage}-storage-bucket-01392334
iamRoleStatements:
    - Effect: Allow
    Action:
        - dynamodb:DescribeTable
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
    Resource: "arn:aws:dynamodb:us-east-1:*:*"

functions:
events:
    handler: handler.events
    events:
    - http:
        method: GET
        path: /events

resources:
Resources:
    events:
    Type: AWS::DynamoDB::Table
    Properties:
        TableName: ${self:provider.environment.EVENTS_TABLE}
        StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
        AttributeDefinitions:
        - AttributeName: id
            AttributeType: S
        KeySchema:
        - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
        ReadCapacityUnits: 1
        WriteCapacityUnits: 1
    storage:
    Type: AWS::S3::Bucket
    Properties:
        BucketName: ${self:provider.environment.STORAGE_BUCKET}

Answer 1

The libcurl-devel & openssl-devel packages are missing, use one of the lambci/lambda Docker images to build and make sure all the dependencies are installed.

Change your serverless.yml to:

provider:
    runtime: python2.7
    [...]

custom:
  pythonRequirements:
    dockerizePip: true
    dockerFile: ./Dockerfile

Then add a Dockerfile with the following contents:

FROM lambci/lambda:build-python2.7

ENV PYCURL_SSL_LIBRARY=openssl

RUN yum install -y libcurl-devel openssl-devel