Testing Python AWS Lambda boto3 initialization

Question

For lambda, it's best practice to initialise dependencies outside the handler.

I am creating a simple python function that works like the blueprints:

import boto3


s3 = boto3.client('ssm')


def lambda_handler(event, context):
    # some code here

And the test

from lambda_function import handler # Option 1
import lambda_function              # Option 2

class TestHandler(unittest.TestCase):

    @patch('lambda_function.handler.boto3.client')
    def test(self, boto3_mock):
        # ...

I can't seem to properly setup a mock so that the boto.client call doesn't error out with You must specify a region.

On Option 1 it errors out during import call, and on Option 2 it does so during the patch setup

I can't use a ~/.aws/config because it will be used on a CI that can't have that. I also don't wan't to change the boto.client call to include a default region.

Is there something I am missing?

Answer 1

While I'm not sure what the issue is with the above code, I'd advise you to use the moto library when you are trying to mock AWS services in Python ( https://github.com/spulec/moto ):

import boto3
from moto import mock_s3
from lambda_function import handler

class TestHandler(unittest.TestCase):

    @mock_s3
    def test(self, boto3_mock):
       # setup your connection to s3 or ssm. Make sure this matches the setup in the lambda file.
       conn = boto3.client('s3')
       conn.create_bucket(Bucket='mybucket') # setup your fake resources
       # call your lambda function

In addition - and as a somewhat personal preference - I would advise against putting too much logic in your actual lambda function. Just take the incoming event send it to other functions/classes as much as possible. This should help simplify testing.

If you really want to keep using @patch instead of the moto library, I got the code to work with the following:

from mock import patch
from example import lambda_function

class TestStringMethods(unittest.TestCase):

    @patch('example.lambda_function.boto3.client')
    def test_my_model(self, some_mock):
        # further setup of test
        lambda_function.my_handler(None, None)

Here, lambda_function is the file containing your handler, and it is located in the directory/package example . You could also mock boto3 itself with 'example.lambda_function.boto3' and return a client yourself.

Answer 2

I had same issue with boto3 s3 client in my client class and moto in my pytest. I resolved it by wrapping boto3 client into a singleton:

This is my client code hello_world/app.py

class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
    if cls not in cls._instances:
        cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
    return cls._instances[cls]

class S3BotoClient(metaclass=Singleton):
    def __init__(self, *args):
        print(f"creating s3 cient with args {args}")
        self.client = boto3.client("s3", *args)

def lambda_handler(event, context):
    s3 = S3BotoClient().client
    s3.list_buckets()

And this is a unit test:

from moto import mock_s3
from hello_world import app

@pytest.fixture()
def apigw_event():
    # return some sample event here

@mock_s3
def test_lambda_handler(apigw_event):
    ret = app.lambda_handler(apigw_event, "")
    ret = app.lambda_handler(apigw_event, "")
    # do assertions here

So S3 client is instantiated only once and after moto virtual env is initialized