Is KCL for AWS kinesis processing thread safe?

Question

We have an application which process the data from kinesis and maintains some state for few seconds.we are afraid whether the maintained state can be affected by the multithreaded nature of KCL.

Can anybody tell us whether RecordProcessor from KCL is thread safe?.

Answer 1

KCL is a wrapper library around your custom logic that processes your records. The purpose of the library is to manage the Kinesis side of things while you focus on the record processing logic. KCL will align you EC2 workers to a certain shard or shards (usually 1 EC2 worker to 1 shard) and maintain a DynamoDB table which stores the sequencing keys.

Your custom application logic is responsible for maintaining state and thread-safety.

By default, a list of Kinesis records (target size is defined by you) that you have picked up from your shard is passed to your code to be processed. You can do this sequentially or fork them to threads if you wish. Not until you return from this processing method will KCL request more records from the shard for you.