How do I use Grequests on AWS Lambda?

Question

I have a Lambda function in Python that uses several libraries with binary dependencies: numpy , scipy , Pillow , etc. In the past, I have successfully compiled them on Amazon Linux & deployed to Lambda .

I recently added Grequest to the pile of dependencies, though, and it's throwing errors that don't seem direclty related to Grequest . Other questions on this topic have resulted in dead-ends, so putting my hat in the ring.

The first error was a simple Grequests requires gevent or something similar. To solve this, I tried running pip install gevent --no-binary :all: on an Amazon Linux instance, bundling that with my code and uploading to Lambda. This had no effect on the error.

I then downloaded the src from the gevent repo and compiled it using make (the commands in the repo's README threw errors I didn't record). This produced an egg file, which I converted to a whl file & ran pip install gevent.whl -t . .

I bundled the resulting code with my lambda.zip and uploaded. This led to a new error : module initialization error: lib/libptf77blas.a.gz: invalid ELF header

libptf77blas.a.gz is a file in the lib folder in my lambda.zip . This folder contains several .so and .a files, which I built on AWS Linux while assembling numpy , scipy , Pillow , etc.

As far as I know, this is a dependency for numpy . The part I don't understand is: my function used numpy (and presumably libptf77blas.a.gz just fine prior to adding grequests .

So I assume something about compiling gevent broke the dependencies for my other binary -dependant libs, or gevent itself requires libptf77blas.a.gz and is confused at what it finds.

I ran ldd on the .so files gevent depends on, but none referenced libptf77blas.a.gz . But, I'm a compiling newb, so that's hardly conclusive.

I'd like to achieve the parallel upload Grequests enables, but am unclear how to fix this, or which alternative libraries to use (I usually write JS, so Python is a second language).

Answer 1

The problem was entirely a stupid user error from an unrelated matter--follow the steps in the OP and you should be fine.

Or just use the copy of grequests I posted on github . It's a zip with grequests , gevent , and greenlet compiled on an Amazon Linux instance. It works in my Lambda Function just fine.

Answer 2

As the other questions have stated, you need to make sure the binaries for libev and other things, are compiled using the Amazon Linux Image . Basically, you need to install things like numpy, and Grequests on a machine running Amazon Linux as they use C code that needs to be compiled.

The tutorial link in that question looks quite good for building things from source, albeit a little bit more than you need for this.

If you're that new to compiling binaries you should read a bit about make, and look at some examples like building curl . Since you're on Amazon Linux, it uses yum and is quite similar to CentOS.

You might be able to get away with installing numpy and Grequests in a venv on a docker image in CentOS and zipping up the contents from that similar to the tutorial, but I'd first go with launching an EC2 instance using amazon's image and installing stuff there like the tutorial does.