[英]How to install Numpy and Pandas for AWS Lambdas?
Problem: I wanted to use Numpy and Pandas in my AWS lambda function.问题:我想在我的 AWS lambda 函数中使用 Numpy 和 Pandas。 I am working on Windows 10 with PyCharm.我正在使用 PyCharm 在 Windows 10 上工作。 My function compiles and works fine on local machine, however, as soon as package it up and deploy on AWS, it breaks down giving errors in importing the numpy and pandas packages.我的函数在本地机器上编译并运行良好,但是,一旦将其打包并部署在 AWS 上,它就会在导入 numpy 和 pandas 包时出现错误。 I tried reinstalling both packages and then redeploying however, error remained the same.我尝试重新安装两个软件包,然后重新部署,但是错误仍然相同。
StackOverFlow Solutions: Other people are having similar issues and fellow users have suggested that this is mainly compatibility issue, because Python libraries are compiled on Windows whereas, AWS Lambda runs on linux machines. StackOverFlow 解决方案:其他人也有类似的问题,其他用户表示这主要是兼容性问题,因为 Python 库是在 Windows 上编译的,而 AWS Lambda 在 linux 机器上运行。
Question: What's the best way to create a deployment package for AWS on windows 10?问题:在 Windows 10 上为 AWS 创建部署包的最佳方法是什么? Is there a way I can specify targeted platform while installing packages through PIP.有没有办法在通过 PIP 安装软件包时指定目标平台。 Apparently there is an option in pip with tag --platform but I cannot figure out how to use it.显然 pip 中有一个带有标签 --platform 的选项,但我不知道如何使用它。 Any helps?有帮助吗?
Like often there is more than one way to come to a solution.就像通常有不止一种方法可以找到解决方案。
The preferred way imho is to use AWS lambda layers, because it separates the functional code from the dependencies.恕我直言,首选方式是使用 AWS lambda 层,因为它将功能代码与依赖项分开。 The basics are explained here . 这里解释了基础知识。
python
.将依赖项放在名为python
的文件夹中。I also had a similar question (how to use numpy within a lambda function).我也有一个类似的问题(如何在 lambda 函数中使用 numpy)。
James provided a recent answer that now makes this much easier than before: AWS now provides a "native" (ie AWS provided) layer for SciPy that you can simply configure to be added as a layer when you define your function. James提供了一个最近的答案,现在比以前更容易了:AWS 现在为 SciPy 提供了一个“本机”(即 AWS 提供的)层,您可以简单地将其配置为在定义函数时添加为一个层。
Putting a link to his answer here, for people that come across this thread first (like me)在这里放置一个指向他的答案的链接,对于首先遇到这个线程的人(像我一样)
...So while you still use layers, you no longer have to build/maintain/install your own SciPy layer yourself, but simply use the one provided by AWS. ...因此,当您仍然使用层时,您不再需要自己构建/维护/安装您自己的 SciPy 层,而只需使用 AWS 提供的层。
So a much better solution now.所以现在有一个更好的解决方案。
Both Numpy and Pandas are available as public layers here: https://github.com/keithrozario/Klayers Numpy 和 Pandas 都可以作为公共层使用: https : //github.com/keithrozario/Klayers
As you mention, it's difficult to build this on Windows, typically you'd need a Linux system to build the python requirements, or build it in a docker container.正如您提到的,在 Windows 上构建它很困难,通常您需要一个 Linux 系统来构建 python 要求,或者在 docker 容器中构建它。
I'm not a Windows 10 user, but I'm guessing you can use WSL (Windows Subsystem for Linux) to build the requirements locally and zip them up before using them as layers.我不是 Windows 10 用户,但我猜你可以使用 WSL(Linux 的 Windows 子系统)在本地构建需求并在将它们用作层之前将它们压缩。 To save trouble though, I'd rather just use the public layers I mentioned earlier.不过,为了省事,我宁愿只使用我之前提到的公共层。
Full disclosure: I own that repo that published the layers.完全披露:我拥有发布这些层的仓库。 It's a free project --but with one downside.这是一个免费项目——但有一个缺点。 I delete older layers if a new version of the package (or one of it's dependencies) are upgraded.如果升级了包的新版本(或其依赖项之一),我会删除旧层。 Generally speaking, if you use the latest layer version, you're guaranteed at least 30 days before the layer will be deleted.一般来说,如果您使用最新的图层版本,则可以保证至少 30 天后该图层将被删除。 Functions that use deleted layers will still work, but you'd be unable to deploy new versions of functions on old layers.使用已删除图层的函数仍然可以使用,但您将无法在旧图层上部署新版本的函数。
Hope that clears things up.希望这能说明问题。
What you need is compressing the codes then upload them.您需要的是压缩代码然后上传它们。
zip -r9 ../function.zip .
zip -g function.zip function.py
aws lambda update-function-code --function-name python37 --zip-file fileb://function.zip
(python37 is the function name here) aws lambda update-function-code --function-name python37 --zip-file fileb://function.zip
(python37 是这里的函数名)
to use zip
commands, the easiest way is to using cygwin or use Windows Subsystem for Linux , but as zip
command is just a command tool to compress files, any GUI compressing tool should work too.要使用zip
命令,最简单的方法是使用cygwin或使用Windows Subsystem for Linux ,但由于zip
命令只是一个压缩文件的命令工具,任何 GUI 压缩工具也应该可以工作。
Amazon created a repository that deals with your situation: https://github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/blank-python亚马逊创建了一个处理您情况的存储库: https : //github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/blank-python
The blank app is an example on how to push a lambda function that depends on requirements, with the bonus that being made by Amazon.空白应用程序是关于如何推送依赖于需求的 lambda 函数的示例,亚马逊提供了奖励。
One concern: it uses bash scripts, therefore you'll need to adapt those or use WSL to get it work.一个问题:它使用 bash 脚本,因此您需要调整这些脚本或使用 WSL 使其工作。 (I'm pretty confident that its doable on windows, most of the work is done by the aws cli) (我非常有信心它在 Windows 上是可行的,大部分工作都是由 aws cli 完成的)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.