Exception when loading values from Azure Key Vault via ConfigurationBuilder

Question

I'm 100% certain this was working earlier today but there doesn't seem to be any reported service outages on https://status.azure.com/en-gb/status .
I've checked the URL and the Key Vault resource and it look fine. I haven't yet tried it locally (need to request permissions from tech-ops!)
The app is running as an app service on Azure using a Linux docker image.

Builder code:

var secretClient = new SecretClient(new Uri(keyVaultUri), new DefaultAzureCredential());
builder.AddAzureKeyVault(secretClient, new KeyVaultSecretManager());

Stack trace (some details redacted, such as the key vault URL):

2022-01-13T16:45:42.786111197Z  ---> Azure.RequestFailedException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786118197Z  ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786122397Z  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
2022-01-13T16:45:42.786126397Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-01-13T16:45:42.786130397Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-01-13T16:45:42.786134397Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-01-13T16:45:42.786138697Z    at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
2022-01-13T16:45:42.786142497Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786146297Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786149997Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786153897Z    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786157797Z    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786162097Z    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786165998Z    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-01-13T16:45:42.786169998Z    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
2022-01-13T16:45:42.786173998Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786177798Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786181398Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786185198Z    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786197598Z    at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786201798Z    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786205698Z    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786209498Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786213198Z    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786216898Z    at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
2022-01-13T16:45:42.786220598Z    at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
2022-01-13T16:45:42.786224398Z    at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken)
2022-01-13T16:45:42.786228398Z    at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
2022-01-13T16:45:42.786232298Z    at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
2022-01-13T16:45:42.786236698Z    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
2022-01-13T16:45:42.786240398Z    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
2022-01-13T16:45:42.786244098Z    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
2022-01-13T16:45:42.786247998Z    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
2022-01-13T16:45:42.786251798Z    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
2022-01-13T16:45:42.786255598Z    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
2022-01-13T16:45:42.786259299Z    at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
2022-01-13T16:45:42.786262999Z    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
2022-01-13T16:45:42.786266599Z    at ****.Core.ConfigurationHelper.GetConfiguration(IEnumerable`1 lstInMemorySettings)
2022-01-13T16:45:42.786270299Z    at ****.Core.ConfigurationHelper.GetConfiguration()
2022-01-13T16:45:42.786273999Z    at ****.Core.ConfigurationHelper.GetSerilogLogger()
2022-01-13T16:45:42.786277599Z    at ****.Core.ConfigurationHelper.SetupSerilog()
2022-01-13T16:45:42.786281299Z    at ****.Core.NoDatabaseMicroServiceProgramBase`1.StartApplication(String[] lstArgs)
2022-01-13T16:45:42.786288499Z    at ****.IEModule.GatewayApi.Service.Program.Main(String[] lstArgs) in /src/****.IEModule.GatewayApi.Service/Program.cs:line 9
2022-01-13T16:45:42.786292599Z  ---> (Inner Exception #1) Azure.RequestFailedException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786296399Z  ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786300199Z  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
2022-01-13T16:45:42.786303799Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-01-13T16:45:42.786307699Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-01-13T16:45:42.786311799Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-01-13T16:45:42.786315799Z    at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
2022-01-13T16:45:42.786319699Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786323499Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786327099Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786330899Z    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786334999Z    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786338999Z    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786342799Z    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-01-13T16:45:42.786346699Z    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
2022-01-13T16:45:42.786350599Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786354300Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786357900Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786361600Z    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786371300Z    at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786375400Z    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786379400Z    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
2022-01-13T16:45:42.786383300Z 
2022-01-13T16:45:42.786386800Z  ---> (Inner Exception #2) Azure.RequestFailedException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786390600Z  ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786394400Z  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
2022-01-13T16:45:42.786398200Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-01-13T16:45:42.786402000Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-01-13T16:45:42.786405800Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-01-13T16:45:42.786409900Z    at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
2022-01-13T16:45:42.786413600Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786417500Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786421200Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786425000Z    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786428800Z    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786432600Z    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786436400Z    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-01-13T16:45:42.786440500Z    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
2022-01-13T16:45:42.786444400Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786448100Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786451801Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786458401Z    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786462401Z    at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786466101Z    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786469901Z    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
2022-01-13T16:45:42.786473701Z 
2022-01-13T16:45:42.786477201Z  ---> (Inner Exception #3) Azure.RequestFailedException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786481001Z  ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable (****.vault.azure.net:443)
2022-01-13T16:45:42.786484801Z  ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
2022-01-13T16:45:42.786488501Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-01-13T16:45:42.786492301Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-01-13T16:45:42.786496001Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-01-13T16:45:42.786499901Z    at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
2022-01-13T16:45:42.786503701Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786507601Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786511401Z    at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
2022-01-13T16:45:42.786515201Z    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786519001Z    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786522801Z    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-01-13T16:45:42.786526601Z    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-01-13T16:45:42.786530501Z    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
2022-01-13T16:45:42.786537401Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786541301Z    --- End of inner exception stack trace ---
2022-01-13T16:45:42.786544802Z    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
2022-01-13T16:45:42.786548502Z    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786552302Z    at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786556002Z    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
2022-01-13T16:45:42.786559802Z    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
2022-01-13T16:45:42.793386973Z 

Any ideas?

Answer 1

not a clue why it was broken but deleting the web app resource from Azure and recreating it worked. Maybe the identity became corrupt after a previous deployment?

Working now anyway: :)