如何减少 Azure AKS IP 地址占用？

Question

After upgrading to AKS v. 1.24 .network type: Azure CNI), finding that kube-system pods take up most of the IP space, leaving very few IPs for application deployments.

For example, here are the kube-system pods running on one of the VMs:

NAMESPACE                NAME                                                   READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES
kube-system              ama-logs-m5rrq                                         2/2     Running   0          25d   10.65.197.16   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              ama-logs-rs-c4f77bf75-cpvgl                            1/1     Running   0          25d   10.65.197.24   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              azure-ip-masq-agent-rcc7m                              1/1     Running   0          25d   10.65.197.15   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              azure-npm-lg7f6                                        1/1     Running   0          25d   10.65.197.15   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              cloud-node-manager-h7w25                               1/1     Running   0          25d   10.65.197.15   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              coredns-59b6bf8b4f-bp9nb                               1/1     Running   0          25d   10.65.197.17   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              csi-azurefile-node-bdpwc                               3/3     Running   0          8d    10.65.197.15   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              konnectivity-agent-85c86bd7ff-wsj5k                    1/1     Running   0          25d   10.65.197.23   aks-agentpool-45621317-vmss00000b   <none>           <none>
kube-system              kube-proxy-hzxhl                                       1/1     Running   0          8h    10.65.197.15   aks-agentpool-45621317-vmss00000b   <none>           <none>

Since we deploy to private.networks, this is becoming a challenge to keep provisioining bigger.networks and keep increasing max pods per IP.

Any ideas how one can free up some of this space?

Answer 1

If the AKS cluster created with the Azure CNI plugin , all the pods in the Kube.netes cluster make use of IPs part of Virtual.network which we created for the AKS cluster.

All cluster Nodes and pods will become part of the cluster su.net (10.224.0.0/16). So, if we are hosting 1000 pods on the AKS cluster, it will pick 1000 Ips from the su.net of .NET. Check the below pods IP details:

NAMESPACE  NAME  READY  STATUS  RESTARTS  AGE  IP  NODE  NOMINATED NODE  READINESS GATES
default  nginxd-67d79c7976-7zv7v  1/1  Running  0  4h19m  10.224.0.17  aks-agentpool-20514180-vmss000000  <none>  <none>
default  nginxd-67d79c7976-kfwwj  1/1  Running  0  4h19m  10.224.0.86  aks-agentpool-20514180-vmss000000  <none>  <none>
default  nginxd-67d79c7976-rt5j9  1/1  Running  0  4h19m  10.224.0.55  aks-agentpool-20514180-vmss000000  <none>  <none>
default  nginxd-67d79c7976-s8pms  1/1  Running  0  4h19m  10.224.0.108  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  azure-ip-masq-agent-sfkzv  1/1  Running  0  5h  10.224.0.4  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  cloud-node-manager-5td8p  1/1  Running  0  5h  10.224.0.4  aks-agentpool-20514180-vmss000000 <none>  <none>
kube-system  coredns-59b6bf8b4f-j7fwq  1/1  Running  0  5h1m  10.224.0.112  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  coredns-59b6bf8b4f-tcsc7  1/1  Running  0  5h  10.224.0.96  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  coredns-autoscaler-5655d66f64-8tddq  1/1  Running  0  5h1m  10.224.0.56  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  csi-azuredisk-node-p92rv  3/3  Running  0  5h  10.224.0.4  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  csi-azurefile-node-fn8pw  3/3  Running  0  5h  10.224.0.4  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  konnectivity-agent-66c54fd7cd-9trm7  1/1  Running  0  4h48m  10.224.0.102  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  konnectivity-agent-66c54fd7cd-c96sp  1/1  Running  0  4h48m  10.224.0.88  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  kube-proxy-dkxpp  1/1  Running  0  5h  10.224.0.4  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  metrics-server-7dd74d8758-j2lph  2/2  Running  0  5h  10.224.0.46  aks-agentpool-20514180-vmss000000  <none>  <none>
kube-system  metrics-server-7dd74d8758-rh266  2/2  Running  0  5h  10.224.0.104  aks-agentpool-20514180-vmss000000  <none>  <none>  

All the services will take IP from the service CDIR. Check the services Ips below:

NAMESPACE  NAME  TYPE  CLUSTER-IP  EXTERNAL-IP  PORT(S)  AGE
default  kubernetes  ClusterIP  10.0.0.1  <none>  443/TCP  5h6m
default  nginx-service  LoadBalancer  10.0.174.204  20.246.132.249  80:32229/TCP  4h20m
kube-system  kube-dns  ClusterIP  10.0.0.10  <none>  53/UDP,53/TCP  5h5m
kube-system  metrics-server  ClusterIP  10.0.5.36  <none>  443/TCP  5h5m

Cross-check Networking details in.networking of AKS once provisioned.
AKS cluster > Networking:

• If you are choosing Azure-CNI, have to plan IP spaces accordingly.

• To prevent pods from make use of real IP addresses use kub.net plugin or azure-cni-overlay.network plugin.

• azure-cni-overlay-plugin is in preview state and available in west central us and east us regions.

• If you choose to select kub.net plugin, it will create a.network for us with default values and it will create a logical IP spaces to make use of pods in the cluster.

• Those Ips are reusable across multiple clusters.
  User can see the.network details in.networking tab once AKS provisioned successfully.

Default values of AKS cluster for kub.net.networking plugin as below:

Here, when pods created in the cluster, it will pic IPs from the Pod CIDR. This is the main difference between Azure-CNI and Kub.net plugin.

Pod CIDR means logical grouping of IPs which is not part of the VNet which we create during AKS cluster.

Check Pod assigned IPs below. It will match with Pod CIDR range:

NAMESPACE  NAME  READY  STATUS  RESTARTS  AGE  IP  NODE  NOMINATED NODE  READINESS GATES
default  nginxd01-784cf56f68-db6wz  1/1  Running  0  4h26m  10.244.0.13  aks-agentpool-97773100-vmss000000  <none>  <none>
default  nginxd01-784cf56f68-fqtjn  1/1  Running  0  4h26m  10.244.0.12  aks-agentpool-97773100-vmss000000  <none>  <none>
default  nginxd01-784cf56f68-mj4w9  1/1  Running  0  4h26m  10.244.0.14  aks-agentpool-97773100-vmss000000  <none>  <none>
default  nginxd01-784cf56f68-vwx84  1/1  Running  0  4h26m  10.244.0.11  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  azure-ip-masq-agent-mgdbr  1/1  Running  0  4h51m  10.224.0.4  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  cloud-node-manager-7zg2b  1/1  Running  0  4h51m  10.224.0.4  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  coredns-59b6bf8b4f-hh8l5  1/1  Running  0  4h50m  10.244.0.8  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  coredns-59b6bf8b4f-rsmmb  1/1  Running  0  4h52m  10.244.0.5  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  coredns-autoscaler-5655d66f64-nb2kk  1/1  Running  0  4h52m  10.244.0.6  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  csi-azuredisk-node-lm4mf  3/3  Running  0  4h51m  10.224.0.4  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  csi-azurefile-node-p5mws  3/3  Running  0  4h51m  10.224.0.4  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  konnectivity-agent-5f4cf5c7dd-r85ht  1/1  Running  0  4h10m  10.244.0.15  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  konnectivity-agent-5f4cf5c7dd-vg6xn  1/1  Running  0  4h10m  10.244.0.16  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  kube-proxy-m8n8x  1/1  Running  0  4h51m  10.224.0.4  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  metrics-server-7dd74d8758-6lqgs  2/2  Running  0  4h50m  10.244.0.10  aks-agentpool-97773100-vmss000000  <none>  <none>
kube-system  metrics-server-7dd74d8758-s4sl9  2/2  Running  0  4h50m  10.244.0.9  aks-agentpool-97773100-vmss000000  <none>  <none>

Ensure that Pod CIDR range must not overlap with any other IPs in the.network. Kub.net.network plugin uses Route table also.

If the user needs more flexibility in controlling IP address space go for the Azure-cli-overlay. But it is in the preview state and does not recommend for production workloads.

Azure-CNI-overlay reference : azure-CNI-overlay.network