如何在 kubernetes 中划分 n 个 pod 之间的空间

Question

We are using Kubernetes and we need to do "Smart partitioning" of data. We want to split the space between 1 to 1000 between n running Pods, And each pod should know which part of the space is his to handle (for pooling partitioned tasks).

So for example, if we have 1 pod he will handle the whole space from 1-1000.

When we scale out to 3 pods, each of them will get the same share.

Pod 1 - will handle 1-333

Pod 2 - 334-667

Pod 3 667-1000

Right now the best way that we find to handle this issue is to create a Stateful-set, that pooling the number of running pods and his instance number and decide which part of the space he needs to handle. Is there a smarter/built-in way in Kubernetes to partition the space between nodes in this manner?

Service fabric has this feature built-in.

Answer 1

There are NO native tools for scaling at the partition level in K8s yet.

Only custom solutions similar to what you have came up with in your original post.

Answer 2

Provide another customized way for doing this for your reference. Based on this tech blog of Airbnb

Given the list of pods and their names, each pod is able to deterministically calculate a list of partitions that it should work on. When we add or remove pods from the ReplicaSet, the pods will simply pick up the change, and work on the new set of partitions instead

How do they do is based on the their repo . I summarized the key components here (Note: the repo is written in Java).

1. Get how many pods running in the k8s namespace, and sort by pod name ( code ). Code snippet

String podName = System.getenv("K8S_POD_NAME");
String namespace = System.getenv("K8S_NAMESPACE");
NamespacedKubernetesClient namespacedClient = kubernetesClient.inNamespace(namespace);
ReplicaSet replicaSet;

// see above code link to know how to get activePods, remove it here because it is too long

int podIndex = activePods.indexOf(podName);
int numPods = activePods.size();

2. Every time you call the above code, you will have deterministic list of podIndex and numPods . Then, using this information to calculate the range this pod is responsible for

List<Integer> partitions = new ArrayList<>();
int split = spaceRange / numPods;
int start = podIndex * split;
int end = (podIndex == numPods - 1) ? spaceRange - 1 : ((podIndex + 1) * split) - 1;
for (int i = start; i <= end; i++) {
  partitions.add(i);
}

3. Since the number of pods will be changed anytime, you may need a executorService.scheduleWithFixedDelay to periodically update the list as here

executorService.scheduleWithFixedDelay(this::updatePartitions, 0, 30, TimeUnit.SECONDS);

This approach is not the best, since if you set scheduleWithFixedDelay with 30 seconds, any pod change won't be captured within 30 seconds. Also, it is possible in a short period of time, two pods may be responsible for the same space, and you need to handle this special case in your business logics as Airbnb tech blog does.