AWS Lambda：如何给 ffmpeg 大文件？

Question

Scenario:

Using AWS Lambda (Node.js), I want to process large files from S3 ( > 1GB). The /tmp fs limit of 512MB means that I can't copy the S3 input there. I can certainly increase the Lambda memory space, in order to read in the files.

• Do I pass the memory buffer to ffmpeg? (node.js, how?)
• Or....should I just make an EFS mount point and use that as the transcoding scratchpad?

Answer 1

You can just use the HTTP(s) protocol as input for ffmpeg.

Lambda has max 10GB memory limit, and data transfer speed from S3 is around 300MB per second the last time I test. So if you have only 1GB max video and are not doing memory intensive transformation, this approach should work fine

ffmpeg -i "https://public-qk.s3.ap-southeast-1.amazonaws.com/sample.mp4" -ss 00:00:10 -vframes 1 -f image2 "image%03d.jpg"

Answer 2

ffmpeg works on files, so maybe an alternative would be to setup a unix pipe and then read that pipe with ffmpeg, constantly feeding it with the s3 stream.

But maybe you'd wanna consider running this as an ECS task instead, you wouldn't have a time constraint, and not the same storage constraint either. Cold start of it using Fargate would be 1-2 minutes though, which maybe isn't acceptable?

Answer 3

Lambda now supports up to 10Gb storage:

https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/

Update with cli: $ aws lambda update-function-configuration --function-name PDFGenerator --ephemeral-storage '{"Size": 10240}'