AKS 连接到 Azure su.net 中的私有 .net 和 terraform

Question

I would like to connect an AKS cluster to an internal .NET/su.net using terraform. Ideally i would like a single IP to be assigned to the service and for all pods in that service to maintain a private service IP. Something like:

• .net/su.net IP cidr to be 172.16.0.0/16
• service cidr to be 172.16.1.0/24
• pods to be on 10.10.10.0/24

where essentially a single IP gets allocated to service cidr. this way we can maintain separation between the pods and the outside world. I would then have an ingress controller setup what would allow external access, and if the pods need to access resources on the internal .net, they should be able to be routed through the service --> su.net etc.

I can't see to get this to work. What's the best way of configuring this in terraform?

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kube.netes_cluster.network_profile

to be clear i would not like each pod to receive su.net address allocated.

Answer 1

The you can use kub.net as the CNI. Kub.net uses a so called Overlay Network and is doing therefore NAT, this means AKS cluster nodes gets IPs inside your Su.net and the Pods receive an IP address from a logically different address space.

I honestly would recommend to user bigger.networks (at least /16) as the ones you specified:

NOTE: IF the .net & Su.net already exist you need to import them with Terraform to create your AKS there.

  network_profile {
    network_plugin     = kubenet
    pod_cidr           = 10.10.10.0/24
    service_cidr       = 172.16.1.0/24
    docker_bridge_cidr = 172.18.0.1/16
    dns_service_ip     = 172.16.1.10
    outbound_type      = userDefinedRouting
  }

  default_node_pool {
    vnet_subnet_id     = azurerm_subnet.YOUR_SUBNET.id
    ...
  }

Egress:

Kube.netes will route egress traffic not from the service to the su.net as you described/want it. By default AKS will use a LoadBalancer with a PublicIP . Alternatively you can specifyuserDefinedRouting as outbound type for your AKS and create own routes.

Ingress:

Ingress will be routed to the AKS via an Azure LoadBalancer. The Ingress controller will get a Public IP that you can also create in advance with terraform (only the PublicIP, not the LoadBalancer) and assign to the Service for the Ingress Controller.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup # only needed if the LB is in another RG
  name: ingress-nginx-controller
spec:
  loadBalancerIP: <YOUR_STATIC_IP>
  type: LoadBalancer

You can also create an internal LoadBalancer and assign a private IP from your su.net:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  name: ingress-nginx-controller-internal
spec:
  loadBalancerIP: <YOUR_STATIC_PRIVATE_IP>
  type: LoadBalancer