Unable to import module 'lambda_function': No module named 'flatten_json'

Question

Gettting the below error while running the lambda code, I am using the library called

from flatten_json import flatten

I tried to look for a lambda layer, but did not find any online, please let me know if any one used this before or suggest any alternative

Answer 1

flatten_json library is missing.

Use pip install flatten_json to get it

Answer 2

There are four steps you need to do:

1. Download the dependency.
2. Package it in a ZIP file.
3. Create a new layer in AWS .
4. Associate the layer with your Lambda .

My answer will focus on 1. and 2. as they are what is most important to your problem. Unfortunately, packaging Python dependencies can be a bit more complicated than for other runtimes.

The main issue is that some dependencies use C code under the hood, especially performance critical libraries, for example for Machine Learning etc.

C code needs to be compiled and if you run pip install on your machine the code will be compiled for your computer. AWS Lambdas use a linux kernel and amd64 architecture. So if you are running pip install on a Linux machine with AMD or Intel processor, you can indeed just use pip install . But if you use macOS or Windows, your best bet is Docker.

Without Docker

pip install --target python flatten_json
zip -r layer.zip python

With Docker

The lambci project provides great Docker container for building and running Lambdas. In the following example I am using their build-python3.8 image.

docker run --rm -v $(pwd):/var/task lambci/lambda:build-python3.8 pip install --target python flatten_json
zip -r layer.zip python

Be aware that $(pwd) is meant to be your current directoy. On macOS and WSL this should work, but if it does not work you can just replace it with the absolute path to your current directory.

Explanation

Those commands will install the dependency into a target folder called python . The name is important, because it is one of two folders of a layer where Lambda looks for dependencies .

The python folder is than archived recursively ( -r ) in a file called layer.zip .

Your next step is to create a new Layer in AWS and associated your function with that layer.

Answer 3

There are two options to choose from

Option 1) You can use a deployment package to deploy your function code to Lambda.

• The deployment package (For eg zip) will contain your function's code and any dependencies used to run the function's code.
• Hence, you can package flatten_json as your code to the Lambda.
• Check Creating a function with runtime dependencies page in aws documentation, it explains the use-case of having requests library. In your scenario, the library would be flatten_json

Option 2) Create a layer that has the library dependencies you need, in your case just flatten_json . And then attach that layer to your Lambda.

• Check creating and sharing lambda layers guide provided by AWS.

How to decide between 1) and 2)?

• Use Option 1) when you just need the dependencies in just that one Lambda. No need to create an extra step of creating a layer.
• Layers are useful if you have some common code that you want to share across different Lambdas. So if you need the library accessible in other Lambdas as well, then it's good to have a layer[ Option 2) ] that can be attached to different lambdas.

Answer 4

You can do this is in a Lambda if you don´t want to create the layer. Keep in mind it will run slower since it has to install the library in every run:

import sys
import subprocess
subprocess.call('pip install flatten_json -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
import flatten_json