在 AWS SAM 中运行 pytest 不会在模板中使用环境变量。yaml

Question

I'm testing my lambda with pytest. My lambda, GetDevicesFunction , connects to a database using a method from a shared module, aurora , located in the utils_layer/python directory. The parameters for the database connection come from template.yaml . Everything works fine when I run sam build && sam local invoke but when I run my pytest , the environment vars don't seem to get pulled in from my template. Is this expected or am I missing something?

Here is my project...

# project structure

├── __init__.py
├── get_devices
│   ├── __init__.py
│   ├── app.py
│   ├── requirements.txt
├── template.yaml
├── tests
│   ├── __init__.py
│   └── unit
│       ├── __init__.py
│       └── test_handler.py
└── utils_layer
    ├── __init__.py
    ├── python
    │   ├── __init__.py
    │   ├── aurora.py
    │   ├── pg8000

# template.yaml

Globals:
  Function:
    Runtime: python3.8
    MemorySize: 256
    Timeout: 60
    Layers:
      - !Ref UtilsLayer
  Environment:
      Variables:
        DATABASE_NAME: !FindInMap [ResourcesName, !Ref MyEnvironment, databaseName]
        DATABASE_HOST: !FindInMap [ResourcesName, !Ref MyEnvironment, databaseHost]
        DATABASE_PORT: !Ref DatabasePort
        DATABASE_USER: !FindInMap [ResourcesName, !Ref MyEnvironment, databaseUser]
        DATABASE_PASSWORD: !FindInMap [ResourcesName, !Ref MyEnvironment, databasePassword]  
        ENVIRONMENT: !Ref MyEnvironment 

Mappings:
  ResourcesName:
    dev:      
      databaseHost: <db_host>
      databaseName: <db_name>
      databaseUser: <db_user>
      databasePassword: <db_password>

Parameters:
  MyEnvironment:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - staging
      - prod

Resources:
  GetDevicesFunction:
    Type: AWS::Serverless::Function
    Properties:      
      CodeUri: get_devices/
      Handler: app.lambda_handler                                                  
      Events:
        GetDevicesApiEvent:
          Type: Api
          Properties:         
            Path: /devices
            Method: GET   
            
  UtilsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      Description: Utils layer
      ContentUri: utils_layer/
      CompatibleRuntimes:        
        - python3.8
      RetentionPolicy: Delete 

# aurora.py

import pg8000

db_host = os.environ.get('DATABASE_HOST')
db_port = os.environ.get('DATABASE_PORT')
db_name = os.environ.get('DATABASE_NAME')
db_user = os.environ.get('DATABASE_USER')
db_password = os.environ.get('DATABASE_PASSWORD')

def make_conn():   
    conn = None
    try:                                          
        conn = pg8000.connect(
            database=db_name, 
            user=db_user, 
            password=db_password, 
            host=db_host                
            )  
    except Exception as e:
        print(f'Connection error: {e}')
    return conn

# test_handler.py

import pytest
from get_devices import app

def test_lambda_handler(apigw_event, mocker):        
    ret = app.lambda_handler(apigw_event, "")  
    data = ret['body']['data'] 
    assert ret["statusCode"] == 200    
    assert len(data) > 0 

Any suggestions on how to pull in env vars from my template would be appreciated.

Answer 1

I did something like this in conftest.py :

import os
import yaml

# Read contents from template file. put all parameters out into envs
try:
    with open('template.yaml', 'rt') as handle:
        # Add constructor to handle !'s
        def any_constructor(loader, tag_suffix, node):
            if isinstance(node, yaml.MappingNode):
                return loader.construct_mapping(node)
            if isinstance(node, yaml.SequenceNode):
                return loader.construct_sequence(node)
            return loader.construct_scalar(node)
        # Add constructors
        yaml.add_multi_constructor('', any_constructor, Loader=yaml.SafeLoader)

        # Load yaml
        template = yaml.safe_load(handle)

        # Add to env
        for param in template['Parameters'].items():
            key = param[0]
            value = param[1]['Default']
            os.environ[key] = value

except yaml.YAMLError as e:
    raise ValueError(e)

Answer 2

You can use the same env.json intended to invoke the function locally . In this case I use one specifically for test called test.env.json (excluded from source control in .gitignore ) with the environment parameters, with this format:

{
    "GetDevicesFunction": {
      "DATABASE_NAME": "dbname",
      "DATABASE_HOST": "dbhost",
      "DATABASE_PORT": "5432",
      "DATABASE_USER": "dbuser",
      "DATABASE_PASSWORD": "dbpassword",
      "ENVIRONMENT": "?"
    }
}

and use a fixture to load them:

from . import get_devices

import os
import json
import pytest


ENVIRONMENT_PATH = os.path.join(
    os.path.dirname(os.path.abspath(get_devices.__file__)),
    "../test.env.json"
)


@pytest.fixture()
def test_environ():
    """Load environment variables to mock"""
    data = {}
    with open(ENVIRONMENT_PATH) as json_file:
        data = json.load(json_file)
    for (k, v) in data["GetDevicesFunction"].items():
        os.environ[k] = str(v)
    return data


def test_lambda_handler(apigw_event, mocker, test_environ):   
    from get_devices import app
    ret = app.lambda_handler(apigw_event, "")  
    data = ret['body']['data'] 
    assert ret["statusCode"] == 200    
    assert len(data) > 0 

If you have used sam build --use-container , and your database is launched locally in a docker container bound to the default network, you can also use the test.env.json file to invoke the function locally calling:

sam local invoke GetDevicesFunction --env-vars test.env.json --docker-network docker_default --debug

and with sam build you can:

sam local invoke GetDevicesFunction --env-vars test.env.json --debug