Unable to import numpy 1.19.1 in AWS Lambda No module named 'numpy.core._multiarray_umath'

Question

I am unable to import numpy 1.19.1 in AWS Lambda with python3.8 on AWS Lambda

I am using the following dependencies:

• pandas 1.1.0
• pyarrow 1.0.0
• numpy 1.19.1
• psycopg2 2.8.5

Because I work on a windows environment, I created an EC2 Linux instance installed python3.8 and downloaded all required libraries, then I added them into the project, but the moment I try to import pandas I get the following:

[ERROR] ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "/var/lang/bin/python3.8",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.2" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "/var/task/src/py38-lib-test.py", line 28, in py38test
    import pandas
  File "/tmp/lib/pandas/__init__.py", line 16, in <module>
    raise ImportError(END RequestId: 07762380-1fc4)

Lastly, I noticed AWS Lambda provides a layer with numpy and sci-kit, I tried removing my numpy version but kept the rest and added the layer to the function, but the same error occurs.

Thanks in advance your comments.

Answer 1

Is your ec2 instance an amazon linux2 machine? You could also try building and run a docker image for amazon linux 2 and get the python libs compatible to the environment you need in your Lambda, by volume mounting to your host.

Something similat to docker lambda:

https://github.com/lambci/docker-lambda/tree/master/python3.8

Answer 2

AWS Lambda function don't work this way. If you open the Pandas package it'll be having the Numpy package with them but they would not work. The easy solution is to first download the required packages separately depending upon your python version and work enviroment from this site, unzip them and add them to your project directory. Create a.zip of your project and deploy it on AWS Lambda function. It'll work this way. You can refer to this site in order to follow the complete procedure.

Answer 3

I use the layer provided by Klayers to solve the problem.

Suppose you're running python 3.8 in us-east-1 region, according to this Klayers document , you can use arn:aws:lambda:ap-east-1:770693421928:layer:Klayers-python38-numpy:19 as your layer so that you can run import numpy in the lambda function.

Answer 4

I had the same issue, tried packaging all libs with my base code, tried custom lambda layer by separating numpy and pandas libs. Nothing worked.

Used default AWS Layers. In the default layers, AWS provides layers like AWSSDKPandas, CodeGuru, Lambda Insights, etc. AWSSDKPandas layer is packaged with pandas libs and other dependencies like numpy, etc.

So I removed numpy dependency from my base package and added AWSSDKPandas as Lambda layer. Worked well.