Retry pattern with configurable handling parameters for exceptions in .NET

Question

I need to enhance basic Retry pattern implementation for handling multiple types of exceptions. Say I want to implement a method (M) that re-attempts some action. And if that action causes an exception, main method catches it and passes to some exception evaluator (E). Now, the responsibility of the "E" is to return back an appropriate wait period to its caller (method M), who eventually enforces this delay. The "E" should also take into account the attempt for each type of occurred exception. For instance, "M" called 2 times on ConnectionLostException , and 3 times on "DatabaseInaccessibleException" I found similar, although not identical question here .

I did basic implementation that works without "E" method:

    public enum IntervalGrowthRate { None, Linear, Exponential, Random };

    public static async Task<T> RetryAsync<T>(
    Func<Task<T>> action,
    IDictionary<string, (IntervalGrowthRate, int, int)> retrySettings) {

    int waitMs = 0;
    int totalAttempts = 0;
    Exception lastException = null;
    IDictionary<string, int> retryAttempts = new Dictionary<string, int>();

    while (true) { 
        try {
            await Task.Delay(waitMs);
            return await action().ConfigureAwait(false);
        } catch (Exception ex) {
            var exceptionName = ex.GetType().FullName;

            if (retrySettings.TryGetValue(exceptionName, out var settings)) {
                var intervalRate = settings.Item1;
                var retryInterval = settings.Item2;
                var retryCount = settings.Item3;

                lastException = ex;
                retryAttempts.TryGetValue(exceptionName, out int currentAttempt);
                retryAttempts[exceptionName] = ++currentAttempt;

                if (currentAttempt <= retryCount) {
                    waitMs = CalculateDelay(intervalRate, retryInterval, currentAttempt);
                    Logging.LogError("Hit an exception and will retry: {0}", activityId, ex.ToString());
                    totalAttempts++;
                } else break;
            }
            else throw;
        }
    }

    var exceptionMessage = string.Format($"{action.Method.Name} method execution failed after retrying {totalAttempts} times.");
    throw new Exception(exceptionMessage, lastException);
}

 private static int CalculateDelay(IntervalGrowthRate growthRate, int delayMs, int currentAttempt) {
    // No delay necessary before the first attempt
    if (currentAttempt < 1) {
        return 0;
    }

    switch (growthRate){
        case IntervalGrowthRate.Linear :
            return delayMs * currentAttempt;
        case IntervalGrowthRate.Exponential :
            return delayMs * (int)Math.Pow(2, currentAttempt);
        case IntervalGrowthRate.Random :
            return (int)(delayMs * currentAttempt * (1 + new Random().NextDouble()));
        case IntervalGrowthRate.None :
        default :
            return delayMs;
    };
}

But the problem is that I need a more flexible logic for exception evaluation. Say look for keywords in an exception message, check InnerException, etc. Any help is appreciated!

EDIT: This is how to call the Retry:

var settings = new Dictionary<string, (IntervalGrowthRate, int, int)>()
{
    ["System.DivideByZeroException"] = (IntervalGrowthRate.Exponential, 1000, 2),
    ["System.OverflowException"] = (IntervalGrowthRate.Linear, 3000, 3)
};

var task = await RetryAsync(
    async () => 
    {
        // do something that can trigger an exception
    },
    settings
);

Answer 1

I found a solution. Probably not the most perfect one, but it's easy to implement. So the main idea is to have retry settings in a separate class and track retry attempts independently for each object:

/// <summary>
/// Class to consolidate retry logic
/// </summary>
public class RetryToken
{
    /// <summary>
    /// The growth function of timeout interval between retry calls
    /// </summary>
    public enum Backoff { None, Linear, Exponential };
    
    private readonly int _maxAttempts;

    private readonly TimeSpan _minWaitTime;

    private readonly Backoff _backoffMode;

    private readonly Func<Exception, bool> _shouldRetry;

    private int _currentRetryAttempt = 0;

    public RetryToken(int maxAttempts, TimeSpan minWaitTime, Backoff backoffMode = Backoff.None, Func<Exception, bool> shouldRetry = null)
    {
        _maxAttempts = maxAttempts;
        _minWaitTime = minWaitTime;
        _backoffMode = backoffMode;
        _shouldRetry = shouldRetry;
    }

    /// <summary>
    /// Checks whether the token knows how to handle the exception
    /// </summary>
    /// <param name="ex">The exception</param>
    /// <returns><c>true</c> if the token can handle the exception, otherwise <c>false</c></returns>
    public bool CanHandle(Exception ex) {
        return _shouldRetry == null || _shouldRetry(ex);
    }

    /// <summary>
    /// Checks if the token has any retry attempts left
    /// </summary>
    /// <returns><c>true</c> if the token has retry attempts (active), otherwise <c>false</c> (inactive)</returns>
    public bool IsActive() {
        return _currentRetryAttempt < _maxAttempts;
    }

    /// <summary>
    /// Simulates retry attempt: increments current attempt and returns 
    /// wait time associated with that attempt.
    /// </summary>
    /// <returns>The wait time for the current retry</returns>
    public TimeSpan GetTimeoutDelay() {
        return CalculateDelay(_backoffMode, _minWaitTime, ++_currentRetryAttempt);
    }
    
    /// <summary>
    /// Calculates the delay needed for the current retry attempt
    /// </summary>
    /// <param name="backoffMode">Growth rate of the interval</param>
    /// <param name="startInterval">Initial interval value (in ms)</param>
    /// <param name="currentAttempt">The current retry attempt</param>
    /// <returns>Wait time</returns>
    private static TimeSpan CalculateDelay(Backoff backoffMode, TimeSpan delayTime, int currentAttempt) {
        // No delay necessary before the first attempt
        if (currentAttempt < 1) {
            return TimeSpan.Zero;
        }

        switch (backoffMode){
            case Backoff.Linear :
                return TimeSpan.FromTicks(delayTime.Ticks * currentAttempt);
            case Backoff.Exponential :
                return TimeSpan.FromTicks(delayTime.Ticks * (int)Math.Pow(2, currentAttempt));
            case Backoff.None :
            default :
                return delayTime;
        }
    }
}

Then call these objects from the main RetryAsync method:

/// <summary>
/// Asynchronously retries action that returns value and throws the last occurred exception if the action fails.
/// </summary>
/// <typeparam name="T">The generic type</typeparam>
/// <param name="operation">The transient operation</param>
/// <param name="retryTokens"><see cref="RetryToken" /> objects</param>
/// <returns>Value of the action</returns>
public static async Task<T> RetryAsync<T>(this Func<Task<T>> operation, IEnumerable<RetryToken> retryTokens) {
    TimeSpan waitTime = TimeSpan.Zero;
    var exceptions = new List<Exception>();

    while (true) { 
        try {
            await Task.Delay(waitTime);
            return await operation().ConfigureAwait(false);
        } catch (Exception ex) {
            exceptions.Add(ex);
            var token = retryTokens.FirstOrDefault(t => t.CanHandle(ex));

            if (token == null) {
                throw; /* unhandled exception with the original stack trace */
            } else if (!token.IsActive()) {
                throw new AggregateException(exceptions)
            }

            waitTime = token.GetTimeoutDelay();
            Console.Writeline("Hit an exception and will retry: {0}", activityId, ex.ToString());
        }
    }
}

Finally, this is how to wrap a transient action into retry pattern with tokens:

var retryTokens = new RetryToken[] {
    new RetryToken(
        maxAttempts: 30, 
        minWaitTime: TimeSpan.FromSeconds(3), 
        backoffMode: Backoff.Linear, 
        shouldRetry: exc => { return exc.Message.Contains("server inaccessible"); }),

    new RetryToken(
        maxAttempts: 8,
        minWaitTime: TimeSpan.FromSeconds(4),
        backoffMode: Backoff.Exponential,
        shouldRetry: exc => { return exc.Message.Contains("request throttled") })
};

Func<Task<bool>> executeDatabaseQuery = async () => {
    return await Task.Run(() => someQuery.Launch());
};

await executeDatabaseQuery.RetryAsync(retryTokens, true);