Multiple consumers to the Kinesis Stream using Kinesis Client Library ( KCL 2.x )

Question

I have an application which uses KCL 2.x to consume records from Kinesis, the data present in different shards of stream is in different format and i want to process them seperately using different KCL consumers with different configurations. For example If i have 3 shards in Kinesis stream i will generate 3 different consumers with different configurations for different shards ie 1 consumer per shard. The issue im facing when im creating multiple consumers is if i configure 3 different consumers via java Code then any one of them is occupping the lease lock over all the shards and other consumers fails to get that lease. For example Total shards: 3, Total Consumers configured: 3,

Application logs :

[2020-07-13 18:55:50,549] (LeaseCoordinator-0000) INFO Worker application-test-stream saw 3 total leases, 3 available leases, 1 workers. Target is 3 leases, I have 0 leases, I will take 3 leases (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:397)
[2020-07-13 18:55:50,549] (LeaseCoordinator-0002) INFO Worker application-test-stream saw 3 total leases, 3 available leases, 1 workers. Target is 3 leases, I have 0 leases, I will take 3 leases (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:397)
[2020-07-13 18:55:50,554] (Thread-22) INFO Initialization complete. Starting worker loop. (software.amazon.kinesis.coordinator.Scheduler:238)
[2020-07-13 18:55:50,842] (LeaseCoordinator-0004) INFO Worker application-test-stream saw 3 total leases, 3 available leases, 1 workers. Target is 3 leases, I have 0 leases, I will take 3 leases (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:397)
[2020-07-13 18:55:51,452] (LeaseCoordinator-0000) INFO Worker application-test-stream successfully took 3 leases: shardId-000000000002, shardId-000000000001, shardId-000000000000 (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:203)
[2020-07-13 18:55:51,457] (LeaseCoordinator-0002) INFO Worker application-test-stream failed to take 3 leases: shardId-000000000002, shardId-000000000001, shardId-000000000000 (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:208)
[2020-07-13 18:55:51,757] (LeaseCoordinator-0004) INFO Worker application-test-stream failed to take 3 leases: shardId-000000000002, shardId-000000000001, shardId-000000000000 (software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker:208)

How can i configure my KCL consumer so that consumer will only take lease for the shard allocated to him.

Answer 1

Typically, a Kinesis consumer is meant to be a stream level consumer - as in, every consumer consumes all shards in a stream. In your case, you can potentially ignore/skip the records in your record processor that you don't care about depending on the shard. There is no out-of-the-box configuration option to only consume specific shards.