Advice on local Lambda testing for data pipeline that involves SQS and DynamoDB

Question

Other devs and I are currently testing/building lambda functions for cleaning data that flows from S3 -> SQS -> Data Router Lambda(python), DynamoDB Rules Engine, and then a text processor in Lambda. We're currently working on the AWS platform but I'm trying to test this part of the data pipeline locally.

Ideally simulating S3 and SQS and dumping the zip files and running it through the lambda function. Currently toying with the SAM-CLI and Visual Studio, but nothing's stuck yet. Any tips?

Answer 1

There are several ways you can approach (local) testing of your AWS application:

1. Use unit tests for the different parts of your "pipeline", mocking the other parts like DynamoDB, SQS, etc.
2. Use something like LocalStack .
3. Every developer has their own "developer environment" in AWS. You could for example prefix every resource with the name of the developer ( john_processing_lambda ). You deploy to AWS and run integration tests from your local machine. You can achieve something like this with tools like Terraform , which allow you to "dynamically" name resources and for example add prefixes with the developers name.

Personally, I think running "AWS on your local machine" via Docker containers or tools like LocalStack not really satisfying. We had the best results with a combination of option 1 and option 3. Both have the upside that you can use the same tests in your CI/CD pipeline.

Furthermore, not running in the actual cloud (AWS) always bears the risk of "forgetting" something. Most notably IAM permissions. So everything runs fine on your local machine, but then it does not work on AWS.

Deploying a separate environment for every developer, so that they can play around with the actual resources and run tests directly in AWS, would be my recommendation. This paired with solid unit tests should yield the best results.

The downside of developer environments in AWS is that a developer has to deploy their code to AWS every time they want to test something. So making deployments fast is important. I found that with sufficient experience, you don't need to deploy that often anymore and this becomes less of an issue. Nevertheless, developer satisfaction in your team is important, so make sure to make this as smooth as possible.