Terraform AKS nsg security rules error out

Question

I'm deploying AKS clusters with Terraform and it's working fine. However, when trying to add security rules to the AKS network security group in the automatically created MC* group, it fails with errors such as:

Creating/Updating Network Security Rule "myRule" (NSG "" / Resource Group "MC_terraform-aks-rg_terraform-aks_westeurope"): network.SecurityRulesClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.Network/networkSecurityGroups/securityRules' under resource group 'MC_terraform-aks-rg_terraform-aks_westeurope' was not found."[0m

If I run terraform apply again, it works and the rules get created. I'm thinking there is a race condition somewhere and I even added a null_resource that executes a sleep command for a couple of minutes, but still errors out on the first try.

main.tf

resource "azurerm_kubernetes_cluster" "aks" {
....................................
}

resource "azurerm_network_security_rule" "https" {
  name                        = "myRule"
  priority                    = 101
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "443"
  destination_port_range      = "*"
  source_address_prefixes     = "${var.imperva_ips}"
  destination_address_prefix  = "${azurerm_public_ip.ingress.ip_address}"
  resource_group_name         = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
  network_security_group_name = "${data.external.aks_nsg_name.result.output}"

  #depends_on = ["azurerm_resource_group.aks", "azurerm_mysql_virtual_network_rule.mysql", "helm_release.ingress"]
  depends_on  = [null_resource.delay]
}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "sleep 60"
  }

  depends_on = [helm_release.ingress]
}

# get the auto-generated NSG name 
data "external" "aks_nsg_id" {
  program = [
    "bash",
    "${path.root}/scripts/aks_nsg_name.sh"
  ]

  depends_on = [azurerm_resource_group.aks]
}

The bash script that pulls the NSG name:

#!/bin/bash 
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'

Answer 1

For AKS cluster, it's not recommended to create the NSG rules manually, Azure will create the appropriate rules for you automatically. If you create the rules manually, it could cause problems. See the description below:

A network security group filters traffic for VMs, such as the AKS nodes. As you create Services, such as a LoadBalancer, the Azure platform automatically configures any network security group rules that are needed. Don't manually configure network security group rules to filter traffic for pods in an AKS cluster. Define any required ports and forwarding as part of your Kubernetes Service manifests, and let the Azure platform create or update the appropriate rules. You can also use network policies, as discussed in the next section, to automatically apply traffic filter rules to pods.

So I would not suggest you create the rules yourself. For more details, see AKS Network Security Group . You'd better use the network policy rather than the NSG rules, and on my side, the network policy is more recommended.

Update:

And the error you got shows that it did not find the rules in the node group. As I see, you need to change the command with a group name of your AKS cluster in the bash. The command you use without a group name will list all the NSG in the subscription, it will not find your NSG if there is not only your AKS cluster.