How to limit Kinesis Firehose records processed at once

Question

I broke my lambda, and have fixed it because I am "smrt". Now however, the backlog of messages that needed to be processed have reached about 15,000.

I keep getting "Firehose encountered timeout errors when calling AWS Lambda. The maximum supported function timeout is 5 minutes" errors, because my lambda cannot process 15k records in 5 minutes.

How can I make the retry not send so many records at once?

Answer 1

The lambda event source has a parameter called Batch size. You should set it to the number of records the lambda can process without timing out.

There is another parameter called "Concurrent batches per shard". This controls how many Lambda instances will be provisioned per shard. This can be set to a max of 10

You need to do some math to determine how many shards you need. So, let's say your Lambda can support 100 records without timing out in 5 minutes. You should set batchSize = 100 If you set ConcurrentBatchesPerShard to 10, this means that you can support 100* 10 = 1K records per 5 minutes. Let's say you are getting 5K records per 5 minutes. Then you need to have 5K/1K = 5 shards in Kinesis stream.