No connection is available to service this operation: when using Azure Redis Cache

Question

I have the following code which I use to get information from the cache. I dont know if maybe my app is opening too many connections or just this error is due to a transient failure on azure redis cache.

This is the stack trace

[RedisConnectionException: No connection is available to service this operation: GET UserProfileInformation|globaladmin@xx.onmicrosoft.com] StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl(Message message, ResultProcessor 1 processor, ServerEndPoint server) in c:\\TeamCity\\buildAgent\\work\\3ae0647004edff78\\StackExchange.Redis\\StackExchange\\Redis\\ConnectionMultiplexer.cs:1922 StackExchange.Redis.RedisBase.ExecuteSync(Message message, ResultProcessor 1 processor, ServerEndPoint server) in c:\\TeamCity\\buildAgent\\work\\3ae0647004edff78\\StackExchange.Redis\\StackExchange\\Redis\\RedisBase.cs:80 StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\\TeamCity\\buildAgent\\work\\3ae0647004edff78\\StackExchange.Redis\\StackExchange\\Redis\\RedisDatabase.cs:1431 xx.Utils.SampleStackExchangeRedisExtensions.Get(IDatabase cache, String key) in C:\\Proyectos\\xx\\xx\\Utils\\SampleStackExchangeRedisExtensions.cs:20
xx.Cache.UserProfile.GetUserProfile(String identityname) in C:\\Proyectos\\xx\\xx\\Cache\\UserProfile.cs:22
x.Controllers.UserProfileController.GetPropertiesForUser() in C:\\Proyectos\\xx\\xx\\Controllers\\UserProfileController.cs:16
lambda_method(Closure , ControllerBase , Object[] ) +61
System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +14

And this is the code

   public static Models.UserProfile GetUserProfile(string identityname)
        {
            /// It needs to be cached for every user because every user can have different modules enabled.

            var cachekeyname = "UserProfileInformation|" + identityname;
            IDatabase cache = CacheConnectionHelper.Connection.GetDatabase();
            Models.UserProfile userProfile = new Models.UserProfile();
            object obj = cache.Get(cachekeyname);
            string userProfileString;
            if (obj != null)
            {
                //get string from cache
                userProfileString = obj.ToString();

                //conver string to our object
                userProfile = JsonConvert.DeserializeObject<Models.UserProfile>(userProfileString);
                return userProfile;
            }
            else
            {
                #region Get User Profile from AD
                Uri serviceRoot = new Uri(SettingsHelper.AzureAdGraphApiEndPoint);
                var token = AppToken.GetAppToken();

                ActiveDirectoryClient adClient = new ActiveDirectoryClient(
                 serviceRoot,
                 async () => await AppToken.GetAppTokenAsync());

                string userObjectID = ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier").Value;

                Microsoft.Azure.ActiveDirectory.GraphClient.Application app = (Microsoft.Azure.ActiveDirectory.GraphClient.Application)adClient.Applications.Where(
                    a => a.AppId == SettingsHelper.ClientId).ExecuteSingleAsync().Result;
                if (app == null)
                {
                    throw new ApplicationException("Unable to get a reference to application in Azure AD.");
                }

                string requestUrl = string.Format("https://graph.windows.net/{0}/users/{1}?api-version=1.5", SettingsHelper.Tenant, identityname);
                HttpClient hc = new HttpClient();
                hc.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
                HttpResponseMessage hrm = hc.GetAsync(new Uri(requestUrl)).Result;

                if (hrm.IsSuccessStatusCode)
                {
                    Models.UserProfile currentUserProfile = JsonConvert.DeserializeObject<Models.UserProfile>(hrm.Content.ReadAsStringAsync().Result);

                    //convert object to json string
                    userProfileString = JsonConvert.SerializeObject(currentUserProfile);

                    cache.Set(cachekeyname, userProfileString, TimeSpan.FromMinutes(SettingsHelper.CacheUserProfileMinutes));
                    return currentUserProfile;
                }
                else
                {
                    return null;
                }
                #endregion
            }
        }




public static class SampleStackExchangeRedisExtensions
    {
        public static T Get<T>(this IDatabase cache, string key)
        {
            return Deserialize<T>(cache.StringGet(key));
        }

        public static object Get(this IDatabase cache, string key)
        {
            return Deserialize<object>(cache.StringGet(key));
        }

        public static void Set(this IDatabase cache, string key, object value, TimeSpan expiration)
        {
            cache.StringSet(key, Serialize(value), expiration);
        }

        static byte[] Serialize(object o)
        {
            if (o == null)
            {
                return null;
            }
            BinaryFormatter binaryFormatter = new BinaryFormatter();
            using (MemoryStream memoryStream = new MemoryStream())
            {
                binaryFormatter.Serialize(memoryStream, o);
                byte[] objectDataAsStream = memoryStream.ToArray();
                return objectDataAsStream;
            }
        }

        static T Deserialize<T>(byte[] stream)
        {
            BinaryFormatter binaryFormatter = new BinaryFormatter();
            if (stream == null)
                return default(T);

            using (MemoryStream memoryStream = new MemoryStream(stream))
            {
                T result = (T)binaryFormatter.Deserialize(memoryStream);
                return result;
            }
        }

Questions are: 1. How can I control a connection exception like the one shown, so that the user doesnt get the error and instead it goes to the DB if redis is unavailable? 2. Is there anyway to retry with transient fault handling for azure redis cache?

Answer 1

I believe these are transient errors. I have seen many of these in my application logs before I implemented simple retry logic. I also had quite a few timeouts. Very simple retry logic, plus adding syncTimeout=3000 to redis connection string resolved all these for me.

public object Get(string key)
{
    return Deserialize(Cache.StringGet(key));
}

public object GetWithRetry(string key, int wait, int retryCount)
{
    int i = 0;
    do
    {
        try
        {
            return Get(key);
        }
        catch (Exception)
        {
            if (i < retryCount + 1)
            {
                Thread.Sleep(wait);
                i++;
            }
            else throw;
        }
    }
    while (i < retryCount + 1);
    return null;
}

Answer 2

I'm using Polly in my Cache Repository to retry all operation with this exception. I tried Retry method from Polly, but this wrong decision and now I'm using WaitAndRetry. With this method you can retry operation with some sleep time - with this you can queue operation to Redis

Answer 3

Also Stack Exchange client has in built retry logic where the client will retry itself. Here is some more information about the configuration options. https://azure.microsoft.com/en-us/documentation/articles/cache-faq/#what-do-the-stackexchangeredis-configuration-options-do

Answer 4

As Rouen suggests, these are probably transient connection errors. Here's a full async example using Polly to handle the retries.

        var value = await Policy 
            .Handle<RedisConnectionException>() // Possible network issue 
            .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3)) // retry 3 times, with a 3 second delay, before giving up
            .ExecuteAsync(async () => {
                return await cache.StringGetAsync(key);
            });

Answer 5

将您的redis nuget软件包更新到最新版本，它应该像我的一样解决您的问题！

Answer 6

If you're using Azure's redis cache, check to see if you're on the basic tier. If so, you'll get this message any time Microsoft updates your server (I found this to happen every few weeks).

A sample of my exact error message:

No connection is active/available to service this operation: GET 43da9f64-da42-4281-845b-82a7d2b7f400#Settings; It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout, mc: 1/1/0, mgr: 10 of 10 available, clientName: RDA04A5E790CA0, IOCP: (Busy=1,Free=999,Min=2,Max=1000), WORKER: (Busy=0,Free=32767,Min=2,Max=32767), v: 2.2.4.27433 It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout

The screenshot below, shown after running automated diagnostics in Azure, was the smoking gun. Some recommendation highlights:

• Upgrade to the standard tier or above to resolve the issue.
• Implement a retry policy to hide the issue.
• Set an update schedule to mitigate issue damage.

Answer 7

Hosting on Azure with Azure Cache for Redis - the only change I did to fix this error was to add ,sslprotocols=tls12 at the end of the connection string that's provided by Azure.

I also updated to the latest version of StackExchange.Redis 2.2.88 but I'm not sure if that has any relevance to this. (probably not) 👍

Found the answer in this GitHub post .