[英]How do I use Grequests on AWS Lambda?
I have a Lambda
function in Python
that uses several libraries with binary dependencies: numpy
, scipy
, Pillow
, etc. In the past, I have successfully compiled them on Amazon Linux
& deployed to Lambda
. 我在
Python
中有一个Lambda
函数,该函数使用几个具有二进制依赖性的库: numpy
, scipy
, Pillow
等。过去,我已经在Amazon Linux
上成功编译了它们并将其部署到Lambda
。
I recently added Grequest
to the pile of dependencies, though, and it's throwing errors that don't seem direclty related to Grequest
. 不过,我最近将
Grequest
添加到了一堆依赖项中,并且它抛出的错误似乎与Grequest
。 Other questions on this topic have resulted in dead-ends, so putting my hat in the ring. 关于这个话题的其他问题导致了死胡同,所以我把帽子戴上了。
The first error was a simple Grequests requires gevent
or something similar. 第一个错误是简单的
Grequests requires gevent
或类似的东西。 To solve this, I tried running pip install gevent --no-binary :all:
on an Amazon Linux
instance, bundling that with my code and uploading to Lambda. 为了解决这个问题,我尝试在
Amazon Linux
实例上运行pip install gevent --no-binary :all:
将其与我的代码捆绑在一起并上传到Lambda。 This had no effect on the error. 这对错误没有影响。
I then downloaded the src
from the gevent
repo and compiled it using make
(the commands in the repo's README threw errors I didn't record). 然后,我从
gevent
库下载了src
并使用make
对其进行了编译(存储库README中的命令引发了我未记录的错误)。 This produced an egg
file, which I converted to a whl
file & ran pip install gevent.whl -t .
这产生了一个
egg
文件,我将其转换为whl
文件并运行pip install gevent.whl -t .
. 。
I bundled the resulting code with my lambda.zip
and uploaded. 我将生成的代码与
lambda.zip
捆绑在一起并上传了。 This led to a new error : module initialization error: lib/libptf77blas.a.gz: invalid ELF header
这导致了一个新错误 :
module initialization error: lib/libptf77blas.a.gz: invalid ELF header
libptf77blas.a.gz
is a file in the lib
folder in my lambda.zip
. libptf77blas.a.gz
是我的lambda.zip
中的lib
文件夹中的文件。 This folder contains several .so
and .a
files, which I built on AWS Linux while assembling numpy
, scipy
, Pillow
, etc. 此文件夹包含几个
.so
和.a
文件,这些文件是我在组装numpy
, scipy
, Pillow
等时在AWS Linux上构建的。
As far as I know, this is a dependency for numpy
. 据我所知,这是
numpy
的依赖项。 The part I don't understand is: my function used numpy
(and presumably libptf77blas.a.gz
just fine prior to adding grequests
. 我不明白的部分是:我的函数使用
numpy
(大概libptf77blas.a.gz
在添加grequests
之前就grequests
。
So I assume something about compiling gevent
broke the dependencies for my other binary
-dependant libs, or gevent
itself requires libptf77blas.a.gz
and is confused at what it finds. 因此,我假设有关编译
gevent
打破了我其他与binary
相关的lib的依赖关系,或者gevent
本身需要libptf77blas.a.gz
并对其发现的内容感到困惑。
I ran ldd
on the .so
files gevent
depends on, but none referenced libptf77blas.a.gz
. 我在
gevent
依赖的.so
文件上运行了ldd
,但没有一个引用libptf77blas.a.gz
。 But, I'm a compiling newb, so that's hardly conclusive. 但是,我是一位正在编译的新手,所以这几乎没有定论。
I'd like to achieve the parallel upload Grequests
enables, but am unclear how to fix this, or which alternative libraries to use (I usually write JS, so Python is a second language). 我想实现
Grequests
启用的并行上载,但尚不清楚如何解决此问题,或不清楚要使用哪个替代库(我通常写JS,因此Python是第二语言)。
The problem was entirely a stupid user error from an unrelated matter--follow the steps in the OP and you should be fine. 问题完全是与无关的用户造成的愚蠢用户错误-按照OP中的步骤操作,您应该会很好。
Or just use the copy of grequests
I posted on github . 或者只是使用我在github上发布的
grequests
的副本。 It's a zip
with grequests
, gevent
, and greenlet
compiled on an Amazon Linux instance. 这是一个
zip
其中grequests
在Amazon Linux实例上编译的grequests
, gevent
和greenlet
。 It works in my Lambda Function
just fine. 它可以在我的
Lambda Function
正常工作。
As the other questions have stated, you need to make sure the binaries for libev and other things, are compiled using the Amazon Linux Image . 正如其他问题所述,您需要确保使用Amazon Linux Image编译libev等二进制文件。 Basically, you need to install things like numpy, and Grequests on a machine running Amazon Linux as they use C code that needs to be compiled.
基本上,您需要在运行Amazon Linux的计算机上安装numpy和Grequests之类的东西,因为它们使用需要编译的C代码。
The tutorial link in that question looks quite good for building things from source, albeit a little bit more than you need for this. 该问题中的教程链接对于从源代码构建事物看起来非常不错,尽管比您需要的更多。
If you're that new to compiling binaries you should read a bit about make, and look at some examples like building curl . 如果您不熟悉编译二进制文件,则应阅读一些有关make的知识,并查看一些示例,例如构建curl 。 Since you're on Amazon Linux, it uses yum and is quite similar to CentOS.
由于您使用的是Amazon Linux,因此它使用yum,并且与CentOS非常相似。
You might be able to get away with installing numpy and Grequests in a venv on a docker image in CentOS and zipping up the contents from that similar to the tutorial, but I'd first go with launching an EC2 instance using amazon's image and installing stuff there like the tutorial does. 您也许可以通过在CentOS的docker映像上的venv上的venv中安装numpy和Grequests并将其内容压缩成类似于本教程的方式来摆脱,但是我首先要使用Amazon的映像启动EC2实例并安装东西就像本教程一样。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.