简体繁体 English

从 lambda 到 s3 的上传时间缓慢的原因

[英]Reason for slow upload time to s3 from lambda

原文 2022-06-09 07:50:47 7 1 java/ amazon-web-services/ amazon-s3/ amazon-ec2/ aws-lambda

We have a service on ec2 and we have a requirement to upload many files to s3 bucket but the number of requests is less than maximum configured on s3.When we upload it using the ec2 instance, it uploads each file in almost 200 ms.我们在 ec2 上有一个服务，我们需要将许多文件上传到 s3 存储桶，但请求数量少于 s3 上配置的最大数量。当我们使用 ec2 实例上传它时，它会在将近 200 毫秒内上传每个文件。 The same files with same content length is taking more time on AWS lambda.具有相同内容长度的相同文件在 AWS lambda 上花费更多时间。 Is there any particular reason for increase in time?时间增加有什么特别的原因吗？ I see increase in time for some files and not for others.Some are taking around 3-4sec for same content length.我发现某些文件的时间有所增加，而其他文件则没有。对于相同的内容长度，有些文件需要大约 3-4 秒。 The ec2 instance is c5.large and I have configured 10 GB to AWS lambda function. ec2 实例是 c5.large，我为 AWS lambda 函数配置了 10 GB。 The bucket is in the same region as the lambda function.This time is obtained from the logs by measuring time before the data is to be uploaded and after upload is complete.These files are made from processing of data from database calls inside the application.存储桶与 lambda 函数位于同一区域。此时间是通过测量要上传数据之前和上传完成之后的时间从日志中获得的。这些文件是通过处理应用程序内部数据库调用的数据生成的。

1 个解决方案

Unfortunately, there's no good way around cold-start times.不幸的是，在冷启动时间没有好办法。

You can use provisioned concurrency, but that means paying for always-on Lambdas, and Java's "load classes only when needed" design means that you'll need to write explicit code to "warm up" the execution environment before the first request comes in.您可以使用预置并发，但这意味着要为永远在线的 Lambda 付费，而 Java 的“仅在需要时加载类”设计意味着您需要编写显式代码以在第一个请求进入之前“预热”执行环境.

One thing that might work is to limit the number of concurrent Lambdas.可能有效的一件事是限制并发 Lambda 的数量。 If you barrage Lambda with requests, it will spin up as many execution environments as it can to process those requests.如果您向 Lambda 发起请求，它将启动尽可能多的执行环境来处理这些请求。 So you'll pay the cold-start time for each of those new invocation environments.因此，您将为每个新的调用环境支付冷启动时间。

However, if you use reserved concurrency , you specify the maximum number of concurrent instances.但是，如果您使用预留并发，则指定最大并发实例数。 You will pay the cold-start penalty for each of those instances, but then Lambda will attempt to reuse instances and not spin up new ones.您将为每个实例支付冷启动罚款，但随后 Lambda 将尝试重用实例而不是启动新实例。 It will, however, shut down environments if there aren't any requests (so you're not paying for them, but will incur cold-start times on the next burst).但是，如果没有任何请求，它将关闭环境（因此您无需为它们付费，但会在下一次爆发时产生冷启动时间）。

You can also smooth out bursts by using an SQS queue: add each file to the queue, configure the number of concurrent Lambdas with reserved concurrency, and let them slowly work their way through the queue.您还可以使用 SQS 队列来平滑突发：将每个文件添加到队列中，配置具有保留并发的并发 Lambda 的数量，并让它们慢慢地通过队列。

Lastly, if all you're doing is uploading files, it's worth considering a different implementation language, such as Python.最后，如果您所做的只是上传文件，那么值得考虑使用不同的实现语言，例如 Python。