如何在 Windows 上创建具有多个节点的 Kubernetes 集群

Question

All kubernetes forums and articles ask to work with minikube that gives you only a single-node kubernetes cluster.

What options are available to work with multi node kubernetes cluster on a Windows environment?.

Answer 1

The problem is that Windows node may only act as a worker node . You can only create a hybrid cluster and have Windows workloads running in Windows pods, talking to Linux workloads running in Linux pods.

Intro to Windows support in Kubernetes :

The Kubernetes control plane, including the master components, continues to run on Linux. There are no plans to have a Windows-only Kubernetes cluster The Kubernetes control plane, including the master components, continues to run on Linux. There are no plans to have a Windows-only Kubernetes cluster .

Full list of limitations can be found in official docs

Control Plane limitations :

Windows is only supported as a worker node in the Kubernetes architecture and component matrix. This means that a Kubernetes cluster must always include Linux master nodes, zero or more Linux worker nodes, and zero or more Windows worker nodes.

Resource management and process isolation :

Linux cgroups are used as a pod boundary for resource controls in Linux. Containers are created within that boundary for network, process and file system isolation. The cgroups APIs can be used to gather cpu/io/memory stats. In contrast, Windows uses a Job object per container with a system namespace filter to contain all processes in a container and provide logical isolation from the host. There is no way to run a Windows container without the namespace filtering in place. This means that system privileges cannot be asserted in the context of the host, and thus privileged containers are not available on Windows. Containers cannot assume an identity from the host because the Security Account Manager (SAM) is separate.

Answer 2

On my windows-10 laptop, I used virtualbox to create 2 ubuntu VMs (Each VM -> 3 GB RAM and 50 GB dynamically sized virtual disks). I used microk8s from https://microk8s.io . Very simple one line installation on each VM: sudo snap install microk8s --classic

Follow instructions at https://microk8s.io/docs/clustering .... one VM becomes the master k8s node and the other VM becomes the worker node joined to the master.

Once that is setup, you may want to setup alias like: alias k='microk8s.kubectl'. Then you can simply do: k apply -f

Answer 3

我能够使用 Oracle 虚拟机在我的 Windows 机上创建一个多节点 kubernetes 集群。！

Answer 4

Hope this might help. I created 4 * centos 8 VMs within Virtual Box hosted on Windows 10. Among the 4* VMs, one VM is set up as master and the rest worker nodes.

Below is my step-by-step set up procedure.

1. Preparation

   1.1 Preparation for basic VM template (node-master-centOS-1)

    1.1.1 (Host) Download centOS 8 image (CentOS-8.1.1911-x86_64-dvd1.iso) from http://isoredirect.centos.org/centos/8/isos/x86_64/ 1.1.2 Install Oracle VM Box from https://www.virtualbox.org/wiki/Downloads

   1.2 Create and Configure a template VM (node-master-centOS-1) in VirtualBox 1.2.1 (VM Box) File->Host Network Manager -> Create a Host-only Ethernet Adapter with Manual address (eg 192.168.56.1/24, DHCP server @ 192.168.56.100/24, DHCP range 101-254) 1.2.2 (VM Box) Pre-configure VM instance 1.2.2.1 (VM Box) System (Memory= 4096MB, Boot Order= Hard Disk -> Optical, Processor=2) 1.2.2.2 (VM Box) Storage (delete IDE controller; under SATA controller, add Optical Drive pointing to centOS-8.x.xxxx-arch-dvdx.iso downloaded at step 1.1.1) 1.2.2.3 (VM Box) Network (Adapter 1= Enable, attached to= NAT; Adapter 2 = Enable, attach to = Host-only Adapter, Name= VirtualBox Host-Only Ethernet Adapter.) Note the Adapter 2 created at step 1.2.1 1.2.2.4 (Host) Settings -> Firewall & network Protection -> Advanced Setting -> In-bound rules -> New Rule -> Custom -> All Programs -> Any port & protocol -> Local IP set as 192.168.56.1 (virtualbox host-only adapter) -> remote IP set as a range from 192.168.5 6.2 - 192.168.56.99 (or as needed) 1.2.2.5 (Host) Settings -> Network and Internet -> Network Connections -> Properties for the adapter which has internet connection -> get working DNS address (eg 192.168.1.1) 1.2.2.6 Start VM instance 1.2.3 (Remote VM) Set up network 1.2.3.1 (Remote VM) Settings -> Network -> Ethernet (enp0s3): ipv4 (manual, 10.0.2.20/24, DNS 10.0.2.3) 1.2.3.2 (Remote VM) Settings -> Network -> Ethernet (enp0s8): ipv4 (manual, 192.168.56.20/24, DNS 192.168.1.1 or as obtained at step 1.2.2.5 so that remote VM inherits internet DNS of the host machine) 1.2.3.3 (Remote VM) Terminal -> sudo ifdown(then ifup) Profile_1 (or enp0s3) -> sudo ifdown(then ifup) Profile_2 (or enp0s8) -> systemctl restart network (if not working: systemctl restart NetworkManager.service) 1.2.4 (Remote VM) Set up hostname 1.2.4.1 (Remote VM) hostnamectl set-hostname node-master-centos-1 (ie {node_1}) 1.2.5 Verify Connectivity 1.2.5.1 (Host) Ping: ping 192.168.56.20 (ie {ip_node_1}) success 1.2.5.2 (Host) SSH: ssh root@192.168.56.20 success -> (SSH) wget www.google.com success (indicates network and DNS is working. If DNS at steps 1.2.2.5 and 1.2.3.2 is not set up, DNS may not work although ip-based internet may be working well.

   1.3 Prepare VM environment 1.3.1 Optional (Remote VM SSH) -> yum install vim git wget bzh -> sh -c "$(wget -O- https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh )" (ohmyzsh gives colored scheme to bash) -> vi .zshrc -> change to ZSH_THEME = "bira" -> source .zshrc (this changes the bash color scheme)

    1.3.4 Turn off selinux (Remote VM SSH) -> setenforce 0 -> 1.3.5 Install JDK 8 -> (Remote VM SSH): yum install java-1.8.0-openjdk-devel -> (Remote VM SSH): -> vi /etc/profile, add "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64" and "export PATH=$JAVA_HOME/bin:$PATH" -> source /etc/profile (to avoid duplicated path setting, better skip this step, if 1.3.6 is to be performed) -> (Remote VM SSH): to verify, run javac -version; java -version; which javac; which java; echo $JAVA_HOME; echo $PATH; 1.3.6 Install Apache Maven -> (Remote VM SSH): -> cd /opt -> wget https://www.strategylions.com.au/mirror/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz -> tar xzvf apache-maven-3.6.3-bin.tar.gz -> vi /etc/profile -> add "export PATH=/opt/apache-maven-3.6.3/bin:$PATH" -> source /etc/profile (once is enough) -> to verify, mvn -v 1.3.7 Install Python, Virtual Env, Tensorflow -> (Remote VM SSH) Install Python3 -> yum update -y (update all installed packages) -> yum install gcc openssl-devel bzip2-devel libffi-devel -y -> verify python3: python3 -> (Remote VM SSH) Install VirtualEnv and Tensorflow -> python3 -m venv --system-site-packages ./venv -> source ./venv/bin/activate # sh, bash, or zsh -> pip install --upgrade pip -> pip install --upgrade requests bs4 numpy torch scipy (and so on) -> pip install tenflow==1.15 (tf2.3.x does not work well on my platform) 1.3.8 Install Kubenetes and Docker (Remote VM SSH) -> Turn off selinux -> setenforce 0 -> sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config (sed -i "s/old text/new text/g" file) -> Stop and Disable Firwall -> systemctl stop firewalld -> systemctl disable firewalld -> Disable devices and files for paging and swapping -> swapoff -a -> yes | cp /etc/fstab /etc/fstab_bak ( create a bak file) -> cat /etc/fstab_bak| grep -v swap > /etc/fstab (keep everything back the line with 'swap' to delete swap) -> Re-configure network adaptor -> enable br_netfilter -> vi /etc/modules-load.d/k8s.conf -> insert "br_netfilter" -> modprobe br_netfilter -> set sysctl settings -> vi /etc/sysctl.d/k8s.conf -> net.bridge.bridge-nf-call-ip6tables = 1 -> net.bridge.bridge-nf-call-iptables = 1 -> sysctl --system -> Firwall (k8s use 6443, 2379-2380, 10250-10255 TCP which need to be enabled) -> systemctl enable firewalld -> systemctl start firewalld -> firewall-cmd --permanent --add-port=6443/tcp -> firewall-cmd --permanent --add-port=2379-2380/tcp -> firewall-cmd --permanent --add-port=10250-10255/tcp -> firewall-cmd –reload -> Enable network modules -> vi /etc/sysconfig/modules/ipvs.modules -> insert -> modprobe -- ip_vs -> modprobe -- ip_vs_rr -> modprobe -- ip_vs_wrr -> modprobe -- ip_vs_sh -> modprobe -- nf_conntrack_ipv4 -> modprobe -- ip_vs -> modprobe -- ip_vs_rr -> modprobe -- ip_vs_wrr -> modprobe -- ip_vs_sh -> modprobe -- nf_conntrack_ipv4 -> verify: cut -f1 -d " " /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4 (shows 5 rows) -> Install Kubenetes -> Set up repository -> vi /etc/yum.repos.d/kubernetes.repo, and insert: [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg -> Install K8s -> yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes -> systemctl enable kubelet -> systemctl start kubelet -> systemctl status kubelet (error255) -> journalctl -xe (missing yaml file /var/lib/kubelet/config.yaml which is expected. ) -> Install Docker -> Set up repository -> yum install -y yum-utils -> yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo -> Install & Run Docker -> yum install docker-ce docker-ce-cli containerd.io -> systemctl enable docker -> systemctl start docker -> verify: docker run helloworld -> verify: docker run -it ubuntu bash -> Update Docker Cgroup -> docker info | grep Cgroup (shows cgroup driver: cgroupfs. This needs updated to align with K8s) -> vi /etc/docker/daemon.json, insert: { "exec-opts":["native.cgroupdriver=systemd"] } -> systemctl restart docker -> verify: docker info | grep Cgroup -> Install node.JS and npm -> yum install epel-release (access to the EPEL repository) -> yum install nodejs (it installs nodeJS and npm) -> verify: node --version (v10.21.0) -> verify: npm version (v6.14.4)

   1.4 Create a cluster of 4 VMs by applying the basic template (node-worker-centOS-1,node-worker-centOS-2,node-worker-centOS-3) -> (VM Box): Clone node-master-centOS-1 for three times, each with new MAC -> (Remote VM): update enp0s3 with ipv4 = 10.0.2.21/22/23, respectively. -> (Remote VM): update enp0s8 with ipv4 = 192.168.56.21/22/23, respectively. -> (Remote VM): update hostname = node-worker-centos-1/2/3, respectively. -> (Remote VM SSH): add host mapping (192.168.20.20/21/22/23 node-master/worker-centos-1/2/3) to /etc/hosts for all nodes.

   1.5 Set up Kubernetes Cluster (1* Master, 3* Workers) -> Init Master Node -> (root@node-master-centos-1 ~) kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.20 pod-network-cide=10.244.0.0 is chosen due to the k8s flannel addon used which in its yaml specifies this ip block for pods.

    below results are shown： # kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.20 [init] Using Kubernetes version: v1.20.0 [preflight] Running pre-flight checks [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node-master-centos-1] and IPs [10.96.0.1 192.168.56.20] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost node-master-centos-1] and IPs [192.168.56.20 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost node-master-centos-1] and IPs [192.168.56.20 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 12.004852 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node node-master-centos-1 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)" [mark-control-plane] Marking the node node-master-centos-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: m5ohft.9xi6nyvgu73sxu68 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.56.20:6443 --token m5ohft.9xi6nyvgu73sxu68 \\ --discovery-token-ca-cert-hash sha256:b04371eb9c969f27a0d8f39761e99b7fb88b33c4bf06ba2e0faa0c1c28ac3be0 -> (root@node-master-centos-1 ~) vi /etc/kubernetes/admin.conf, and edit to replace "192.168.56.20" to "node-master-centos-1" (use hostname instead of ip address) -> (root@node-master-centos-1 ~) sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config -> (root@node-master-centos-1 ~) sudo chown $(id -u):$(id -g) $HOME/.kube/config -> (root@node-master-centos-1 ~) kubectl get nodes NAME STATUS ROLES AGE VERSION node-master-centos-1 NotReady control-plane,master 4m3s v1.20.0 -> root@node-master-centos-1 ~) kubeadm token create --print-join-command (to obtain the command to be run on workers) -> By now, the k8s master is initialized which sets pod network to be 10.244.0.0/16 with api server at HTTPS://node-master-centos-1:6443. At this stage, the node-master-centos-1 node is NotReady because Pod Network is not yet deployed which we need to use flannel.yaml (one of addons for podnetwork) -> Join Worker Nodes -> synchronize system time to avoid X509 certification error duruing kubeadm join. Below updates time offsets and adjust systime in one step. -> (root@node-worker-centos-1/2/3 ~) chronyc -a 'burst 4/4' -> (root@node-worker-centos-1/2/3 ~) chronyc -a makestep -> join the worker to cluster -> (root@node-worker-centos-1/2/3 ~) kubeadm join node-master-centos-1:6443 --token cjxoym.okfgvzd8t241grea --discovery-token-ca-cert-hash sha256:b04371eb9c969f27a0d8f39761e99b7fb88b33c4bf06ba2e0faa0c1c28ac3be0 --v=2 -> check node worker status on Master -> (root@node-master-centos-1 ~) kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-master-centos-1 Ready control-plane,master 4h12m v1.20.0 192.168.56.20 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0 node-worker-centos-1 Ready <none> 162m v1.20.0 192.168.56.21 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0 node-worker-centos-2 Ready <none> 142m v1.20.0 192.168.56.22 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0 node-worker-centos-3 Ready <none> 4m41s v1.20.0 192.168.56.23 <none> CentOS Linux 8 4.18.0-147.el8.x86_64 docker://20.10.0 -> (root@node-master-centos-1 ~) kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-74ff55c5b-sfjvd 1/1 Running 0 112m kube-system coredns-74ff55c5b-whjrs 1/1 Running 0 112m kube-system etcd-node-master-centos-1 1/1 Running 0 112m kube-system kube-apiserver-node-master-centos-1 1/1 Running 0 112m kube-system kube-controller-manager-node-master-centos-1 1/1 Running 0 112m kube-system kube-flannel-ds-dmqmw 1/1 Running 0 61m kube-system kube-flannel-ds-hqwqt 1/1 Running 0 2m51s kube-system kube-flannel-ds-qr9ml 1/1 Running 0 22m kube-system kube-proxy-4dpk9 1/1 Running 0 22m kube-system kube-proxy-6tltc 1/1 Running 0 2m51s kube-system kube-proxy-t6k24 1/1 Running 0 112m kube-system kube-scheduler-node-master-centos-1 1/1 Running 0 112m By Now, the kubernetes cluster is set up. As the VMs are not always run, the differences of system time between VMs may cause X509 or other errors. It may be therefore necessary to set up auto-sync scripts runnable on OS startup.