每个 kube.netes 节点一个 pod/job

Question

I would like to schedule periodically kube.netes jobs (with different images)

These jobs are required to run on a node with GPU support (1 GPU device)

Currently If I create two jobs at the same time - the pods will be scheduled both on the same node - while only one pod will have access to GPU device

Is there a way to configure nodes/pods so that scheduler only places one pod per node and once it is completed places next job?

Answer 1

You could set an inter-pod anti-affinity as described in the docs here .

Inter-pod affinity and anti-affinity rules take the form "this Pod should (or, in the case of anti-affinity, should not) run in an X if that X is already running one or more Pods that meet rule Y", where X is a topology domain like node, rack, cloud provider zone or region, or similar and Y is the rule Kubernetes tries to satisfy.

Similar to node affinity are two types of Pod affinity and anti-affinity as follows:

• requiredDuringSchedulingIgnoredDuringExecution
• preferredDuringSchedulingIgnoredDuringExecution

Consider the following Pod spec:

 apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: security operator: In values: - S1 topologyKey: topology.kubernetes.io/zone podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: security operator: In values: - S2 topologyKey: topology.kubernetes.io/zone containers: - name: with-pod-affinity image: registry.k8s.io/pause:2.0

Answer 2

Since you are preferring to use a simple way you can use nodeSelector for selecting your desired node using the node labels and use queueSort for scheduling pods one after another. In short you are defining pods with certain labels to run on a certain node that too on a priority basis. This document gives you a better understanding for achieving your desired functionality.

Answer 3

I ended up using suggestion from @Calum Halpin with Extended Resources for a Node

https://kube.netes.io/docs/tasks/administer-cluster/extended-resource-node/

1. patch the node and add 1 "gpu" to capacity:

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/gpu", "value": "1"}]' \
http://localhost:8001/api/v1/nodes/<node-name>/status

2. Then for your container set resource request for 1 gpu:

    resources:
      requests:
        gpu: 1