Terraform：使用 azurerm_kubernetes_cluster 创建 AKS 集群时指定网络安全组规则

Question

I am using the Terraform azurerm provider version 1.19 to create an AKS cluster. I'd like to specify network security group rules when creating the cluster but I can't figure out how to reference the security group that is created since the generated security group is given a name with random numbers.

Something like:

aks-agentpool-33577837-nsg

Is there a way to reference the created nsg or atleast output the random number used in the name?

Configuration to create the cluster:

resource "azurerm_resource_group" "k8s" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name                = "${var.cluster_name}"
  location            = "${azurerm_resource_group.k8s.location}"
  resource_group_name = "${azurerm_resource_group.k8s.name}"
  dns_prefix          = "${var.dns_prefix}"
  kubernetes_version  = "${var.kubernetes_version}"

  linux_profile {
    admin_username = "azureuser"

    ssh_key {
      key_data = "${file("${var.ssh_public_key}")}"
    }
  }

  agent_pool_profile {
    name    = "default"
    count   = "${var.agent_count}"
    vm_size = "${var.vm_size}"
    os_type = "Linux"
  }

  service_principal {
    client_id     = "${var.client_id}"
    client_secret = "${var.client_secret}"
  }

  tags {
    source      = "terraform"
    environment = "${var.environment}" 
  }
}

This generates a security group which I'd like to add additional rules to. Here's a rule I'd like to add so the nginx-controller's liveness probe can be checked.

resource "azurerm_network_security_rule" "nginx_liveness_probe" {
  name                        = "nginx_liveness"
  priority                    = 100 
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "${var.nginx_liveness_probe_port}"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_kubernetes_cluster.k8s.node_resource_group}"
  network_security_group_name = How do I reference the auto-generated nsg ?
  description = "Allow access to nginx liveness probe"
}

Answer 1

The solution which answers your question:

data "azurerm_resources" "example" {
  resource_group_name = azurerm_kubernetes_cluster.example.node_resource_group

  type = "Microsoft.Network/networkSecurityGroups"
}

output name_nsg {
    value = data.azurerm_resources.example.resources.0.name
}

resource "azurerm_network_security_rule" "example" {
  name                        = "example"
  priority                    = 100
  direction                   = "Outbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "*"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = azurerm_kubernetes_cluster.example.node_resource_group
  network_security_group_name = data.azurerm_resources.example.resources.0.name
}

.. and then in the same way add all your rules.

In general it is better and advised to make use of the automated way Azure Kubernetes Service reacts to the creation of Kubernetes Services, not to use more Terraform (although below Kubernetes yaml can also be used with the Kubernetes Terraform provider ):

A network security group filters traffic for VMs, such as the AKS nodes. As you create Services, such as a LoadBalancer, the Azure platform automatically configures any network security group rules that are needed. Don't manually configure network security group rules to filter traffic for pods in an AKS cluster. Define any required ports and forwarding as part of your Kubernetes Service manifests, and let the Azure platform create or update the appropriate rules. You can also use network policies, as discussed in the next section, to automatically apply traffic filter rules to pods.

It should be possible within the current setup by simple creating a Kubernetes Service to those ports you want to disclose.

For instance, when I deploy a ingress controller, the creation of the Kubernetes Service triggers the creation of an IP address/Loadbalancer with its NSG:

apiVersion: v1
kind: Service
metadata:
  name: ingress-ingress-nginx-controller
  namespace: ingress
spec:
  loadBalancerSourceRanges:
  - 8.8.8.8/32
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

By creating a Kubernetes Service (type LoadBalancer) which maps to the wanted pod port and with loadBalancerSourceRanges specified, a similar setup for your custom destination can be specified.

apiVersion: v1
kind: Service
metadata:
  name: mycustomservice
  namespace: myownnamespace
spec:
  loadBalancerSourceRanges:
  - 8.8.8.8/32 # your source IPs
  - 9.9.9.9/32
  ports:
  - name: myaccessport
    port: 777
    protocol: TCP
    targetPort: mydestinationport
  selector:
    app.kubernetes.io/name: myapp
  type: LoadBalancer

See also the issue in the azurerm provider GitHub

Answer 2

A little late here, but just came across this issue. So, for anyone still looking for a solution, this is what I ended up doing to get the AKS NSG name:

Add this to the *.tf file provisioning your AKS:

resource "azurerm_network_security_rule" "http" {
  name                        = "YOUR_NAME"
  priority                    = 102
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "80"
  destination_port_range      = "*"
  source_address_prefixes     = "${var.ips}"
  destination_address_prefix  = "${azurerm_public_ip.ingress.ip_address}"
  resource_group_name         = "${azurerm_kubernetes_cluster.test.node_resource_group}"
  network_security_group_name = "${data.external.aks_nsg_name.result.output}"

  depends_on = ["azurerm_resource_group.test"]
}

# get the NSG name
data "external" "aks_nsg_name" {
  program = [
    "bash",
    "${path.root}/scripts/aks_nsg_name.sh"
  ]

  depends_on = [azurerm_resource_group.test]
}

Create aks_nsg_name.sh in your project and add the following:

#!/bin/bash 
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks-agentpool | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'

Answer 3

Your AKC is added to an azurerm_resource_group , which you provisioned using Terraform, I assume. If so, you can add a custom azurerm_network_security_group with any number of azurerm_network_security_rule to that resource group, as detailed here .

Example:

resource "azurerm_resource_group" "test" {
  name     = "acceptanceTestResourceGroup1"
  location = "West US"
}

resource "azurerm_network_security_group" "test" {
  name                = "acceptanceTestSecurityGroup1"
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
}

resource "azurerm_network_security_rule" "test" {
  name                        = "test123"
  priority                    = 100
  direction                   = "Outbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "*"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_resource_group.test.name}"
  network_security_group_name = "${azurerm_network_security_group.test.name}"
}

Unfortunately, the name parameter is required on network security group data sources, and wildcards do not seem to be supported, or else that would have been an option as well.