How to cache boto3 API calls in python?

Question

I am working on some code that will interact with the AWS API through boto3, the calls I'm making have big response payloads and it takes a while to process, testing code while developing is getting really frustrating and impractical. Whats the most practical way for me to cache the response of a certain api call , like the one below ?

#!/usr/bin/python3.6
import boto3

client_cw = boto3.client('cloudwatch')
paginator = client_cw.get_paginator('describe_alarms')

for response in paginator.paginate():
    print(response)

Any help will be greatly appreciated.

Answer 1

This is a great situation for using unittest 's mock capability. This allows you to redefine an entire object or a specific behavior of an object. There are two pieces to a solution using this approach:

• You are defining custom behavior for your get_paginator() function, so you would be mocking the boto3.client.get_paginator() method to return a simple custom Paginator class.

• You also need to define a simple custom Paginator class that will only define one method, the one you call in your code, which is paginate() , and when that method is called, it should return a hard-coded paginated list (in JSON format) that reflects whatever you would expect AWS to return.

Here's the modified code:

#!/usr/bin/python3.6
import boto3
from unittest import mock

client_cw = boto3.client('cloudwatch')

# ---
# This is where the mocking starts
with mock.patch("boto3.client") as client:
    # Create a mock paginator to return mock pages
    class MockPaginator(object):
        def paginate(self):
            # Customize this to whatever the "real"
            # paginated list from AWS looks like
            return [
                {
                    "AlarmsList": [
                        {
                            "Name": "my_alarm_1"
                        },
                        {   
                            "Name": "my_alarm_2"
                        }
                    ]
                }
            ]
    # Tell the get_paginator method to return a fake Paginator
    client.get_paginator.return_value = MockPaginator()
    # This is where the mocking ends
    # (but we're still in the mocking context)
    # ---

    # And now on with the show...
    paginator = client.get_paginator('describe_alarms')
    for response in paginator.paginate():
        print(response)

Now, when you call paginator.paginate() , it will return the custom hard-coded JSON list you provide in the custom Paginator class, rather than trying to call up AWS.

Answer 2

I may be late to answer here. But I had a similar use case and ended up creating a project named botocache. You can check it out here :- https://github.com/rams3sh/botocache

It uses the same approach of patching the boto3's specific method using unittest's patch.