Azure Keyvault Bad Request When Called From Kubernetes Pod

Question

I've run into a problem accessing Azure Key Vault whilst converting some services from Microsoft Service Fabric to Kube.netes. Within our ASP.NET core ConfigureServices call, we call AddAzureKeyVault from Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions to inject some sensitive parts of our configuration as shown below.

public void ConfigureServices(IServiceCollection services)
{

    var config = new ConfigurationBuilder()
        .SetBasePath(AppContext.BaseDirectory)
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddAzureKeyVault(
            Environment.GetEnvironmentVariable("KEYVAULT_LOCATION"),
            Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_ID"),
            Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_SECRET")
        ).Build();

        //...
}

This works fine in a docker image running locally, yet once deployed into Azure Kube.netes service the pod is failing with the following...

Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)

Looking at the source of Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions I can reproduce the issue with this minimal code.

public void ConfigureServices(IServiceCollection services)
{
    GetSecrets(
        Environment.GetEnvironmentVariable("KEYVAULT_LOCATION"),
        Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_ID"),
        Environment.GetEnvironmentVariable("KEYVAULT_CLIENT_SECRET")
    ).GetAwaiter().GetResult();
}

public async Task GetSecrets(string loc, string clientId, string clientSecret)
{
    KeyVaultClient.AuthenticationCallback callback =
        (authority, resource, scope) => GetTokenFromClientSecret(authority, resource, clientId, clientSecret);

    IKeyVaultClient _client = new KeyVaultClient(callback);

    //Exception thrown here
    var secrets = await _client.GetSecretsAsync(loc).ConfigureAwait(false);
}

private static async Task<string> GetTokenFromClientSecret(string authority, string resource, string clientId, string clientSecret)
{
    var authContext = new AuthenticationContext(authority);
    var clientCred = new ClientCredential(clientId, clientSecret);
    var result = await authContext.AcquireTokenAsync(resource, clientCred);
    return result.AccessToken;
}

My question is, what is different about this authentication when called from within a pod in AKS, as opposed to a local docker image, that would lead this call to fail?

I've confirmed the pod can access the wider inte.net, the key vault is not firewalled and Insights is showing some Authentication failures in the key vault logs. The Client Id and Secret are correct and have the correct rights since this works locally in Docker. What am I missing?

Answer 1

I just ran your code sample on my machine, local cluster, and an AKS cluster. It's working in all places. Please check your environment. You may be looking in the wrong place. There must be something in your local environment. Try running it in a different environment. For example: from home (without VPN).