[英]AWS Lambda read from SQS without concurrency
My requirement is like this.我的要求是这样的。
I implemented a AWS Lambda to achieve point 1. I have a Lambda which has an sqs trigger.我实施了 AWS Lambda 来实现第 1 点。我有一个 Lambda,它有一个 sqs 触发器。 I have set batch size as 50 and then batch window as 2 hours.
我已将批处理大小设置为 50,然后将批处理 window 设置为 2 小时。 My assumption was that Lambda will get triggered every 2 hours and 50 messages will be delivered to the lambda function in one go and I will create a file for every 50 records.
我的假设是 Lambda 将每 2 小时触发一次,50 条消息将在一个 go 中传递到 lambda function,我将为每 50 条记录创建一个文件。
But I observed that my lambda function is triggered with varied number of messages(sometimes 50 sometimes 20, sometimes 5 etc) even though I have configured batch size as 50.但我观察到我的 lambda function 被不同数量的消息触发(有时 50 有时 20,有时 5 等),即使我将批处理大小配置为 50。
After reading some documentation I got to know(I am not sure) that there are 5 long polling connections which lambda spawns to read from SQS and this is causing this behaviour of lambda function being triggered with varied number of messages.在阅读了一些文档后,我知道(我不确定)有 5 个长轮询连接,其中 lambda 产生以从 SQS 读取,这导致 lambda function 的这种行为被不同数量的消息触发。
My question is我的问题是
A "SQS Trigger" for Lambda is implemented with the so-called Event Source Mapping integration, which polls, batches and deletes messages from the queue on your behalf. Lambda 的“SQS 触发器”通过所谓的事件源映射集成实现,它代表您从队列中轮询、批处理和删除消息。 It's designed for continuous polling, although you can disable it.
它专为连续轮询而设计,但您可以禁用它。 You can set a maximum batch size of up to 10,000 records a function receives ( BatchSize ) and a maximum of 300s long polling time ( MaximumBatchingWindowInSeconds ).
您可以设置最大批量大小为 function 接收的最多 10,000 条记录 ( BatchSize ) 和最长300 秒的长轮询时间 ( MaximumBatchingWindowInSeconds )。 That doesn't meet your once-every-two-hours requirement.
这不符合您每两小时一次的要求。
With the SQS Event Source Mapping integration you can tweak the batch settings, but ultimately the Lambda service is in charge of Lambda scaling.通过 SQS 事件源映射集成,您可以调整批处理设置,但最终 Lambda 服务负责 Lambda 缩放。 As the AWS Blog Understanding how AWS Lambda scales with Amazon SQS standard queues says:
正如 AWS 博客了解 AWS Lambda 如何使用 Amazon SQS 标准队列进行扩展所说:
Lambda consumes messages in batches, starting at five concurrent batches with five functions at a time.
Lambda 批量消费消息,从五个并发批次开始,一次有五个函数。 If there are more messages in the queue, Lambda adds up to 60 functions per minute, up to 1,000 functions, to consume those messages.
如果队列中有更多消息,则 Lambda 每分钟最多添加 60 个函数,最多 1000 个函数来消费这些消息。
You could theoretically restrict the number of concurrent Lambda executions with reserved concurrency , but you would risk dropped messages due to throttling errors .理论上您可以使用预留并发限制并发 Lambda 执行的数量,但您可能会因节流错误而面临丢失消息的风险。
You could try to set the ReservedConcurrency
of the function to 1. That may help.您可以尝试将 function 的
ReservedConcurrency
设置为 1。这可能会有所帮助。 See the docs for reference.请参阅文档以供参考。
A simple solution would be to create a CloudWatch Event Trigger (similar to a Cronjob) that triggers your Lambda function every two hours.一个简单的解决方案是创建一个 CloudWatch 事件触发器(类似于 Cronjob),它每两小时触发一次 Lambda function。 In the Lambda function, you call
ReceiveMessage
on the Queue until you get all messages, process them and afterward delete them from the Queue.在 Lambda function 中,您在 Queue 上调用
ReceiveMessage
直到您获得所有消息,处理它们,然后从 Queue 中删除它们。 The drawback is that there may be too many messages to process within 15 minutes so that's something you'd have to manage.缺点是可能有太多消息需要在 15 分钟内处理,因此这是您必须管理的事情。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.