How does Kinesis distribute shards among workers?

Question

Is there any attempt to keep adjacent shards together when spreading them out over multiple workers? In the documentation example it started with 1 worker/instance and 4 shards. Then auto-scaling occurred and a 2nd worker/instance was started up. The KCL auto-magically moved 2 shards over to worker 2. Is there any attempt at keeping adjacent shards together with a worker when autoscaling? What about when splitting shards?

Thanks

Answer 1

Random.

If you mean "Kinesis Consumer Application" as "Worker", then the consumer application with the most shards loses 1 shard to another application who has less shards.

"Lease" is the correct term here, it describes a consumer application & shard association. And there is not adjacency check for taking leases, it is pure random.

See source code, chooseLeaseToSteal method: https://github.com/awslabs/amazon-kinesis-client/blob/c6e393c13ec348f77b8b08082ba56823776ee48a/src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseTaker.java#L414

Answer 2

Is there any attempt to keep adjacent shards together when spreading them out over multiple workers?

I doubt that's the case. My understanding is that order is maintained only within the boundary of a single key and the boundary of a single key falls within a single shard.

Imagine I have 2 keys, key-a and key-b , and the following events occurred:

["event-1-key-a", "event-2-key-b", "event-3-key-a"]

Now we have 2 events for key-a : ["event-1-key-a", "event-3-key-a"]

and 1 event for key-b : ["event-2-key-b"]

Note that sharding happens exactly like the above -- the 2 events for key-a will always end up in the same shard. With that being the guarantee, maintaining the order among shards is not necessary.