Kinesis如何在工人之间分配碎片？

Question

Is there any attempt to keep adjacent shards together when spreading them out over multiple workers? In the documentation example it started with 1 worker/instance and 4 shards. Then auto-scaling occurred and a 2nd worker/instance was started up. The KCL auto-magically moved 2 shards over to worker 2. Is there any attempt at keeping adjacent shards together with a worker when autoscaling? What about when splitting shards?

Thanks

Answer 1

Random.

If you mean "Kinesis Consumer Application" as "Worker", then the consumer application with the most shards loses 1 shard to another application who has less shards.

"Lease" is the correct term here, it describes a consumer application & shard association. And there is not adjacency check for taking leases, it is pure random.

See source code, chooseLeaseToSteal method: https://github.com/awslabs/amazon-kinesis-client/blob/c6e393c13ec348f77b8b08082ba56823776ee48a/src/main/java/com/amazonaws/services/kinesis/leases/impl/LeaseTaker.java#L414

Answer 2

Is there any attempt to keep adjacent shards together when spreading them out over multiple workers?

I doubt that's the case. My understanding is that order is maintained only within the boundary of a single key and the boundary of a single key falls within a single shard.

Imagine I have 2 keys, key-a and key-b , and the following events occurred:

["event-1-key-a", "event-2-key-b", "event-3-key-a"]

Now we have 2 events for key-a : ["event-1-key-a", "event-3-key-a"]

and 1 event for key-b : ["event-2-key-b"]

Note that sharding happens exactly like the above -- the 2 events for key-a will always end up in the same shard. With that being the guarantee, maintaining the order among shards is not necessary.