k8s running inside CentOS docker container running on my macOS host - swap problems?

Question

I want to run Kubernetes v1.13.5 unit-tests inside a CentOS docker container running on a macOS host. I get errors trying to run kubeadm as it doesn't like having swap enabled.

1) I tried kubeadm init --fail-swap-on=false but get Error: unknown flag: --fail-swap-on . Maybe only supported in older k8s but docs don't seem clear.

2) I tried kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --skip-token-print with /etc/kubernetes/kubeadm-config.yaml like

---
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: ""
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
kubernetesVersion: "v1.13.5"
failSwapOn: false

but that gives... error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster", ... SNIP... error unmarshaling JSON: while decoding JSON: json: unknown field \"failSwapOn\""

3) I tried switching off swap inside my container but I get...

docker run -i -t centos
[root@2ed611b32f1a /]# swapoff -a
swapoff: Not superuser.

or it when privileged mode it ran but didn't affect setup

docker run --privileged -i -t centos 
[root@94f9a0e5e46a /]# swapoff -a
[root@94f9a0e5e46a /]# free -h
              total        used        free      shared  buff/cache   available
Mem:           12Gi       371Mi        11Gi       1.0Mi       1.1Gi        12Gi
Swap:         2.0Gi          0B       2.0Gi

4) I tried flags like docker run --memory 256M --memory-swap 256M but these are not honoured inside the container despitehttps://docs.docker.com/config/containers/resource_constraints/ saying...

PREVENT A CONTAINER FROM USING SWAP

If --memory and --memory-swap are set to the same value, this prevents containers from using any swap. This is because --memory-swap is the amount of combined memory and swap that can be used, while --memory is only the amount of physical memory that can be used.

5) I tried in docker desktop v2.1.0.3 on my laptop adjusting the swap setting via the preferences in its UI, but it won't let me decrease swap below 512.0 MiB.

Can you advise a way to get k8s running inside CentOS docker container running on my macOS host ?

Answer 1

I think the problem may be your apiVersion for kubeadm! You have an old deprecated version in there which is v1alpha3 It is highly recommended here that you migrate your old configuration to use the newer api which is v1beta1

In Kubernetes 1.11 and later, the default configuration can be printed out using the kubeadm config print command. It is recommended that you migrate your old v1alpha3 configuration to v1beta1 using the kubeadm config migrate command, because v1alpha3 will be removed in Kubernetes 1.14.

Answer 2

My 6th (and finally successful) attempt was this. See last 4 lines below.

kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --skip-token-print where kubeadm-config.yaml looks like...

---
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: ""
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
kubernetesVersion: "v1.13.5"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false

Answer 3

If you decided to enable swap, then I'll guess your workloads probably needs it.

I'll just add a smaller disclaimer for readers which are choosing this option as the default solution regardless of there specific workloads.

It seems the Kubelet is not designed to work properly with swap enabled - K8S is very clear about this topic as you can see in the Kubeadm installation :

Swap disabled. You MUST disable swap in order for the kubelet to work properly.

I would recommend reading about Evicting end-user Pods and the relevant features that K8S provides to prioritize memory of pods:

1 ) The 3 qos classes - Make sure that your high priority workloads are running with the Guaranteed (or at least Burstable ) class.

2 ) Pod Priority and Preemption .