Load balancing and scaling in Kinesis Client library (KCL) based application

Question

I am using amazon-kinesis-connectors to build a kinesis client side application. I am figuring out few things about it.

How KCL ensures load balancing and scaling. for example i have a stream with one shard and a lot of records/events are put for 15 mintues in whole day or for certain time. So how it will take care of such sudden traffic and load.

Answer 1

For load balancing, you need multiple shards. Kinesis attempts to balance the number of shards each worker takes on - if you only have one shard, you won't get any benefit from more than one worker. See my related answer here for a bit more information on how Kinesis load balancing works.

As for scaling - each shard has a limit to maximum amount of traffic it can take - from the kinesis limitation docs :

Each shard can support up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second

The way you scale up with kinesis is add more shards - more shards == more throughput. You can either have these shards pre-created, or you can add them as your traffic increases.