"KeyVaultAuthenticationFailure" when Storage Account attempts to Access Customer Managed Key in Key Vault with Private Endpoint (Using Terraform)

Question

We are having trouble creating a Storage Account that uses a Customer Managed Key stored in Key Vault using Terraform. Importantly, the Key Vault sits behind a private endpoint which the Storage Account must use to access.

Getting the following error:

│ Error: updating Customer Managed Key for Storage Account "t3mpnarg47sa01" (Resource >Group "t3-mpn-arg47"): storage.AccountsClient#Update: Failure responding to request: >StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 >Code="KeyVaultAuthenticationFailure" Message="The operation failed because of?>authentication issue on the keyvault." │ │ with >module.example_storage_account.azurerm_storage_account_customer_managed_key.cmk, │ on....\terraform-azurerm-pcs-edited-storage-account\main.tf line 104, in resource >"azurerm_storage_account_customer_managed_key" "cmk": │> 104: resource "azurerm_storage_account_customer_managed_key" "cmk" {

How do we get this connection to work?

The below code shows how we create the key vault, the key vault private endpoint the storage account, and the customer managed key. Creation of .net and su.net not shown:

### Key Vault Creation, Key Vault Private Endpoint Creation, and Networking

resource "azurerm_key_vault" "this" {
  name                = local.name
  resource_group_name = var.resource_group_name
  location            = var.location

  tenant_id = data.azurerm_client_config.current.tenant_id
  sku_name  = var.sku_name

  network_acls {
    default_action             = "Deny"
    bypass                     = "AzureServices"
    ip_rules                   = local.ip_rules #client ip address
    virtual_network_subnet_ids = var.virtual_network_subnet_ids
  }

  enabled_for_deployment          = var.enabled_for_deployment
  enabled_for_disk_encryption     = var.enabled_for_disk_encryption
  enabled_for_template_deployment = var.enabled_for_template_deployment
  enable_rbac_authorization       = true
  purge_protection_enabled        = var.enable_purge_protection
  soft_delete_retention_days      = var.soft_delete_retention_days

  tags = merge(var.tags, local.mandatory_tags)
}

resource "azurerm_role_assignment" "kv_role_admin_kva" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Administrator"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_private_endpoint" "this" {
  name                = local.private_endpoint_name
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = var.subnet_id

  private_dns_zone_group {
    name                 = "privatednszonegroup"
    private_dns_zone_ids = [azurerm_private_dns_zone.this.id]
  }

  private_service_connection {
    name                           = local.private_service_connection_name
    private_connection_resource_id = var.key_vault_id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }

  lifecycle {
    # ignore_changes = [
    #   private_dns_zone_group, # Ignore changes to private_dns_zone_group as it is applied by Azure policy
    # ]
  }

  tags = merge(var.tags, local.mandatory_tags)
}

resource "azurerm_private_dns_zone" "this" {
  name                = "privatelink.vaultcore.azure.net"
  resource_group_name = var.resource_group_name
}

resource "azurerm_private_dns_zone_virtual_network_link" "this" {
  name                  = "vnetlink"
  resource_group_name   = var.resource_group_name
  private_dns_zone_name = azurerm_private_dns_zone.this.name
  virtual_network_id    = var.virtual_network_id
}

data "azurerm_private_endpoint_connection" "this" {
  name                = local.private_endpoint_name
  resource_group_name = var.resource_group_name
  depends_on          = [ azurerm_private_endpoint.this ]
}


### Storage Account Creation & Networking

resource "azurerm_storage_account" "sa" {
  name                = local.name
  resource_group_name = var.resourceGroupName
  location            = data.azurerm_resource_group.arg.location

  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = "LRS"
  access_tier               = "Hot"
  enable_https_traffic_only = true
  is_hns_enabled            = true
  min_tls_version           = "TLS1_2"
  shared_access_key_enabled = false
  allow_blob_public_access  = false

  identity {
    type = "SystemAssigned"
  }

  network_rules {
    default_action             = "Deny"
    bypass                     = ["AzureServices"]
    ip_rules                   = local.ipRules #client ip address
    virtualNetworkSubnetIds =   [module.example_subnet.id]
  }
}

resource "azurerm_key_vault_key" "kvkey" {
  name            = format("cmk-%s", local.name)
  key_vault_id    = var.keyVaultId
  key_type        = "RSA-HSM"
  key_size        = 2048
  key_opts        = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]
  expiration_date = var.expirationDate
}

resource "azurerm_role_assignment" "sa_role_admin_sbdo" {
  scope                = azurerm_storage_account.sa.id
  role_definition_name = "Storage Blob Data Owner"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "sa_role_admin_sqdc" {
  scope                = azurerm_storage_account.sa.id
  role_definition_name = "Storage Queue Data Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "kv_role_client_kvc" {
  scope                = var.keyVaultId
  role_definition_name = "Key Vault Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "kv_role_sa_kvcseu" {
  scope                = var.keyVaultId
  role_definition_name = "Key Vault Crypto Service Encryption User"
  principal_id         = azurerm_storage_account.sa.identity.0.principal_id
}

# Customer Managed Key Creation (fails)
resource "azurerm_storage_account_customer_managed_key" "cmk" {
  storage_account_id = azurerm_storage_account.sa.id
  key_vault_id       = var.keyVaultId
  key_name           = azurerm_key_vault_key.kvkey.name

  depends_on = [
    azurerm_role_assignment.kv_role_client_kvc,
    azurerm_role_assignment.kv_role_sa_kvcseu,
  ]
}

Answer 1

I tried creating a key vault with private endpoint and storage account encryption with customer managed keys with the below code:

You have to set shared_access_key_enabled = true instead of false & key_type = "RSA" instead of RSA-HSM & add necessary depends on on other resource blocks.

You can try with something like below making the required changes:

### Key Vault Creation, Key Vault Private Endpoint Creation, and Networking
provider "azurerm"{
    features{}
}

data "azurerm_virtual_network" "vnet" {
  name = "ansuman-vnet"
  resource_group_name="ansumantest"
}

data "azurerm_client_config" "current" {}
resource "azurerm_key_vault" "this" {
  name                = "terraformtestkv12"
  resource_group_name = "ansumantest"
  location            = "East US"

  tenant_id = data.azurerm_client_config.current.tenant_id
  sku_name  = "standard"

  network_acls {
    default_action             = "Deny"
    bypass                     = "AzureServices"
    ip_rules                   = ["49.xx.xx.234"] #client ip address
    virtual_network_subnet_ids = ["${data.azurerm_virtual_network.vnet.id}/subnets/default"]
  }

  enabled_for_deployment          = true
  enabled_for_disk_encryption     = true
  enabled_for_template_deployment = true
  enable_rbac_authorization       = true
  purge_protection_enabled        = true
  soft_delete_retention_days      = 7
}

resource "azurerm_role_assignment" "kv_role_admin_kva" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Administrator"
  principal_id         = data.azurerm_client_config.current.object_id
}
resource "azurerm_private_dns_zone" "this" {
  name                = "privatelink.vaultcore.azure.net"
  resource_group_name = "ansumantest"
}

resource "azurerm_private_dns_zone_virtual_network_link" "this" {
  name                  = "vnetlink"
  resource_group_name   = "ansumantest"
  private_dns_zone_name = azurerm_private_dns_zone.this.name
  virtual_network_id    = data.azurerm_virtual_network.vnet.id
}
resource "azurerm_private_endpoint" "this" {
  name                = "keyvault-endpoint"
  location            = "eastus"
  resource_group_name = "ansumantest"
  subnet_id           = "${data.azurerm_virtual_network.vnet.id}/subnets/default"

  private_dns_zone_group {
    name                 = "privatednszonegroup"
    private_dns_zone_ids = [azurerm_private_dns_zone.this.id]
  }

  private_service_connection {
    name                           = "keyvault-privatednsconnection"
    private_connection_resource_id = azurerm_key_vault.this.id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }
  depends_on = [
    azurerm_private_dns_zone_virtual_network_link.this
  ]
}

### Storage Account Creation & Networking

resource "azurerm_storage_account" "sa" {
  name                = "ansumantestsa12"
  resource_group_name = "ansumantest"
  location            = "eastus"

  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = "LRS"
  access_tier               = "Hot"
  enable_https_traffic_only = true
  is_hns_enabled            = true
  min_tls_version           = "TLS1_2"
  shared_access_key_enabled = true ##enable to access the storage account with key
  allow_blob_public_access  = false

  identity {
    type = "SystemAssigned"
  }

  network_rules {
    default_action             = "Deny"
    bypass                     = ["AzureServices"]
    ip_rules                   = ["49.xx.xx.234"] #client ip address
    virtual_network_subnet_ids =   ["${data.azurerm_virtual_network.vnet.id}/subnets/default"]
  }
  depends_on = [
    azurerm_key_vault.this,
    azurerm_private_endpoint.this
  ]
}

resource "azurerm_key_vault_key" "kvkey" {
  name            = "storageencryptionkey"
  key_vault_id    = azurerm_key_vault.this.id
  key_type        = "RSA"#as its normal keyvault you can't use RSA-HSM instead use RSA
  key_size        = 2048
  key_opts        = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]
    depends_on = [
    azurerm_key_vault.this,
    azurerm_private_endpoint.this
  ]
}

resource "azurerm_role_assignment" "sa_role_admin_sbdo" {
  scope                = azurerm_storage_account.sa.id
  role_definition_name = "Storage Blob Data Owner"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "sa_role_admin_sqdc" {
  scope                = azurerm_storage_account.sa.id
  role_definition_name = "Storage Queue Data Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "kv_role_client_kvc" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "kv_role_sa_kvcseu" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Crypto Service Encryption User"
  principal_id         = azurerm_storage_account.sa.identity.0.principal_id
}

# Customer Managed Key Creation (fails)
resource "azurerm_storage_account_customer_managed_key" "cmk" {
  storage_account_id = azurerm_storage_account.sa.id
  key_vault_id       = azurerm_key_vault.this.id
  key_name           = azurerm_key_vault_key.kvkey.name

  depends_on = [
    azurerm_role_assignment.kv_role_admin_kva,
    azurerm_role_assignment.kv_role_client_kvc,
    azurerm_role_assignment.kv_role_sa_kvcseu,
    azurerm_storage_account.sa
  ]
}

Output:

Answer 2

I've recently had an issue with creating private endpoints on an azure keyvault using terraform and then was unable to access the resource when in that same terraform I was creating the secrets.

The issue was caused by the private endpoint state was 'pending' and it took several seconds to be 'approved' (using automated rbac)... so just had add a wait in there or put a dependency on the secret creation on a resource you know is created last and takes a while itself to create.