How to update DNS configuration of K8S Pod

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. moluzhui wants to to this question: 这个问题：

My work project requires pod auto-injection (modify DNSConfig, and DNSPolicy as well)

I had a project that wanted to update the DNS configuration of Pod with Operator,

get dns message
get matched pod
modify:
    pod.Spec.DNSConfig = CRD_SPEC
    pod.Spec.DNSPolicy = corev1.DNSNone
client.Update(ctx,&pod)

but when I implemented it, I got the following error

 ERROR   controller-runtime.manager.controller.dnsinjection      Reconciler error        {"reconciler group": "xxxx", "reconciler kind": "xxxxx", "name": "dnsinjection", "namespace": "default", "error": "Pod \"busybox\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 21 identical fields\n  \tPriority:         &0,\n  \tPreemptionPolicy: nil,\n  \tDNSConfig: &core.PodDNSConfig{\n  \t\tNameservers: []string{\n  \t\t\t\"1.2.3.4\",\n- \t\t\t\"0.0.0.0\",\n  \t\t},\n  \t\tSearches: []string{\"ns1.svc.cluster-domain.example\", \"my.dns.search.suffix\"},\n  \t\tOptions:  []core.PodDNSConfigOption{{Name: \"ndots\", Value: &\"2\"}, {Name: \"edns0\"}},\n  \t},\n  \tReadinessGates:   nil,\n  \tRuntimeClassName: nil,\n  \t... // 3 identical fields\n  }\n"}

DNSConfig and DNSPoicy fields are not declared to be unable to be updated in the source code, so why did the update fail?

I got the same error with kubect edit pod busybox and kubectl apply -f modifyed_pod.yml(add DNSConfig) command

I would appreciate it if you could tell me how to solve it.

Answer 1

As the message says, Kubernetes does not support updating most pod's fields directly :

Kubernetes doesn't prevent you from managing Pods directly. It is possible to update some fields of a running Pod, in place. However, Pod update operations like patch , and replace have some limitations:

• Most of the metadata about a Pod is immutable. For example, you cannot change the namespace , name , uid , or creationTimestamp fields; the generation field is unique. It only accepts updates that increment the field's current value.
• If the metadata.deletionTimestamp is set, no new entry can be added to the metadata.finalizers list.
• Pod updates may not change fields other than spec.containers[*].image , spec.initContainers[*].image , spec.activeDeadlineSeconds or spec.tolerations . For spec.tolerations , you can only add new entries.

Why is that?

Pods in Kubernetes are designed as relatively ephemeral, disposable entities :

You'll rarely create individual Pods directly in Kubernetes—even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a controller ), the new Pod is scheduled to run on a Node in your cluster. The Pod remains on that node until the Pod finishes execution, the Pod object is deleted, the Pod is evicted for lack of resources, or the node fails.

Kubernetes assumes that for managing pods and doing any updates you should use a workload resources instead of creating pods directly:

Pods are generally not created directly and are created using workload resources. See Working with Pods for more information on how Pods are used with workload resources. Here are some examples of workload resources that manage one or more Pods:

• Deployment
• StatefulSet
• DaemonSet

You can easily update most fields in workload resources definition and it will work properly. Keep in mind that it won't edit any existing pods - it will delete the currently running pods with old configuration and start the new ones - Kubernetes will make sure that this process goes smoothly :

Modifying the pod template or switching to a new pod template has no direct effect on the Pods that already exist. If you change the pod template for a workload resource, that resource needs to create replacement Pods that use the updated template.

For example, the StatefulSet controller ensures that the running Pods match the current pod template for each StatefulSet object. If you edit the StatefulSet to change its pod template, the StatefulSet starts to create new Pods based on the updated template. Eventually, all of the old Pods are replaced with new Pods, and the update is complete.

Each workload resource implements its own rules for handling changes to the Pod template. If you want to read more about StatefulSet specifically, read Update strategy in the StatefulSet Basics tutorial.

So based on all above information I'd suggest to switch to workload resource, for example deployment :

A Deployment provides declarative updates for Pods and ReplicaSets .

For example - right now I have pod with below definition:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox:1.28
    command:
      - sleep
      - "9999999"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

When I try to run kubectl edit pod busybox command to change dnsPolicy I have the same error as you. However, If I changed to deployment with the same pod definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - image: busybox:1.28
        command:
        - sleep
        - "9999999"
        imagePullPolicy: IfNotPresent
        name: busybox
      restartPolicy: Always
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet

Then if I run kubectl edit deploy busybox-deployment and change dnsPolicy field I will get a new pod with new configuration (the old pod will be automatically deleted).

Keep in mind, if you want to stick with direct pod definition, you can always just delete pod and apply a new, modified yaml as you tried ( kubectl delete pod {your-pod-name} && kubectl apply -f {modified.yaml} ). It will work properly.

Also check:

• Kubernetes Deployments vs StatefulSets
• Workload Resources
• Pod Lifecycle

Answer 2

Like the message says you cannot update a DNS config of a pod: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image .

If you want to inject a DNS config into all pods you need to add the configuration before the pod is created. Look into MutatingAdmissionWebhook as an approach for this.