[英]Polly CircuitBreaker fallback not working
I have the following policies: 我有以下政策:
var retryPolicy = Policy.Handle<Exception>(e => (e is HttpRequestException || e.InnerException is HttpRequestException)).WaitAndRetry(
retryCount: maxRetryCount,
sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
onRetry: (exception, calculatedWaitDuration, retryCount, context) =>
{
Log.Error($"Retry => Count: {retryCount}, Wait duration: {calculatedWaitDuration}, Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}, Exception: {exception}.");
});
var circuitBreaker = Policy.Handle<Exception>(e => (e is HttpRequestException || e.InnerException is HttpRequestException)).CircuitBreaker(maxExceptionsBeforeBreaking, TimeSpan.FromSeconds(circuitBreakDurationSeconds), onBreak, onReset);
var sharedBulkhead = Policy.Bulkhead(maxParallelizations, maxQueuingActions, onBulkheadRejected);
var fallbackForCircuitBreaker = Policy<bool>
.Handle<BrokenCircuitException>()
.Fallback(
fallbackValue: false,
onFallback: (b, context) =>
{
Log.Error($"Operation attempted on broken circuit => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
}
);
var fallbackForAnyException = Policy<bool>
.Handle<Exception>()
.Fallback(
fallbackAction: (context) => { return false; },
onFallback: (e, context) =>
{
Log.Error($"An unexpected error occured => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
}
);
var resilienceStrategy = Policy.Wrap(retryPolicy, circuitBreaker, sharedBulkhead);
var policyWrap = fallbackForAnyException.Wrap(fallbackForCircuitBreaker.Wrap(resilienceStrategy));
public bool CallApi(ChangeMapModel changeMessage)
{
var httpClient = new HttpClient();
var endPoint = changeMessage.EndPoint;
var headers = endPoint.Headers;
if (headers != null)
{
foreach (var header in headers)
{
if (header.Contains(':'))
{
var splitHeader = header.Split(':');
httpClient.DefaultRequestHeaders.Add(splitHeader[0], splitHeader[1]);
}
}
}
var res = httpClient.PostAsync(endPoint.Uri, null);
var response = res.Result;
response.EnsureSuccessStatusCode();
return true;
}
I execute the policy like so: 我像这样执行策略:
policyWrap.Execute((context) => CallApi(changeMessage), new Context(endPoint));
The problem is that I am not getting a hit in the CircuitBreaker callback when an action is executed on open circuit. 问题是在开路执行操作时,CircuitBreaker回调没有受到影响。
I want an API call to be placed through the policy with the exception type to be handled being HttpRequestException
. 我希望通过策略放置API调用,并使用
HttpRequestException
处理异常类型。 Is there something wrong with the policy definitions? 策略定义有问题吗? Why isn't the circuit breaker fallback called?
为什么不调用断路器后备?
I created the following minimum, complete, verifiable example to help explore the problem: 我创建了以下最小,完整,可验证的示例,以帮助探讨问题:
Note: not necessarily a finished product; 注意:不一定是成品; just some minor mods to the posted code and extra annotation, to help explore the question.
只需对发布的代码和额外的注释进行一些小的修改,以帮助探讨问题。
using Polly;
using Polly.CircuitBreaker;
using System;
using System.Net.Http;
using System.Threading.Tasks;
public class Program
{
public static void Main()
{
int maxRetryCount = 6;
double circuitBreakDurationSeconds = 0.2 /* experiment with effect of shorter or longer here, eg: change to = 1, and the fallbackForCircuitBreaker is correctly invoked */ ;
int maxExceptionsBeforeBreaking = 4; /* experiment with effect of fewer here, eg change to = 1, and the fallbackForCircuitBreaker is correctly invoked */
int maxParallelizations = 2;
int maxQueuingActions = 2;
var retryPolicy = Policy.Handle<Exception>(e => (e is HttpRequestException || (/*!(e is BrokenCircuitException) &&*/ e.InnerException is HttpRequestException))) // experiment with introducing the extra (!(e is BrokenCircuitException) && ) clause here, if necessary/desired, depending on goal
.WaitAndRetry(
retryCount: maxRetryCount,
sleepDurationProvider: attempt => TimeSpan.FromMilliseconds(50 * Math.Pow(2, attempt)),
onRetry: (ex, calculatedWaitDuration, retryCount, context) =>
{
Console.WriteLine(String.Format("Retry => Count: {0}, Wait duration: {1}, Policy Wrap: {2}, Policy: {3}, Endpoint: {4}, Exception: {5}", retryCount, calculatedWaitDuration, context.PolicyWrapKey, context.PolicyKey, context.OperationKey, ex.Message));
});
var circuitBreaker = Policy.Handle<Exception>(e => (e is HttpRequestException || e.InnerException is HttpRequestException))
.CircuitBreaker(maxExceptionsBeforeBreaking,
TimeSpan.FromSeconds(circuitBreakDurationSeconds),
onBreak: (ex, breakDuration) => {
Console.WriteLine(String.Format("Circuit breaking for {0} ms due to {1}", breakDuration.TotalMilliseconds, ex.Message));
},
onReset: () => {
Console.WriteLine("Circuit closed again.");
},
onHalfOpen: () => { Console.WriteLine("Half open."); });
var sharedBulkhead = Policy.Bulkhead(maxParallelizations, maxQueuingActions);
var fallbackForCircuitBreaker = Policy<bool>
.Handle<BrokenCircuitException>()
/* .OrInner<BrokenCircuitException>() */ // Consider this if necessary.
/* .Or<Exception>(e => circuitBreaker.State != CircuitState.Closed) */ // This check will also detect the circuit in anything but healthy state, regardless of the final exception thrown.
.Fallback(
fallbackValue: false,
onFallback: (b, context) =>
{
Console.WriteLine(String.Format("Operation attempted on broken circuit => Policy Wrap: {0}, Policy: {1}, Endpoint: {2}", context.PolicyWrapKey, context.PolicyKey, context.OperationKey));
}
);
var fallbackForAnyException = Policy<bool>
.Handle<Exception>()
.Fallback<bool>(
fallbackAction: (context) => { return false; },
onFallback: (e, context) =>
{
Console.WriteLine(String.Format("An unexpected error occured => Policy Wrap: {0}, Policy: {1}, Endpoint: {2}, Exception: {3}", context.PolicyWrapKey, context.PolicyKey, context.OperationKey, e.Exception.Message));
}
);
var resilienceStrategy = Policy.Wrap(retryPolicy, circuitBreaker, sharedBulkhead);
var policyWrap = fallbackForAnyException.Wrap(fallbackForCircuitBreaker.Wrap(resilienceStrategy));
bool outcome = policyWrap.Execute((context) => CallApi("http://www.doesnotexistattimeofwriting.com/"), new Context("some endpoint info"));
}
public static bool CallApi(string uri)
{
using (var httpClient = new HttpClient() { Timeout = TimeSpan.FromSeconds(1) }) // Consider HttpClient lifetimes and disposal; this pattern is for minimum change from original posted code, not a recommendation.
{
Task<HttpResponseMessage> res = httpClient.GetAsync(uri);
var response = res.Result; // Consider async/await rather than blocking on the returned Task.
response.EnsureSuccessStatusCode();
return true;
}
}
}
More than one factor could be causing the fallbackForCircuitBreaker
not to be invoked: 不止一个原因可能导致
fallbackForCircuitBreaker
不被调用:
circuitBreakDurationSeconds
may be set shorter than the overall time taken by the various tries and waits between retries. circuitBreakDurationSeconds
设置为短于重试之间的各种尝试和等待所花费的总时间。 If so, the circuit may revert to half-open state. 如果是这样,则电路可以恢复到半开状态。 In half-open state or closed state , an exception which causes the circuit to break is rethrown as-is.
在半开状态或闭合状态下 ,按原样重新引发导致电路断开的异常。
BrokenCircuitException
is only thrown when an call is prevented from being attempted by a (fully) open circuit. 仅当(完全)开路阻止尝试进行呼叫时,才抛出
BrokenCircuitException
。
Thus, if your circuit has reverted to half-open by the time retries exhaust, the exception thrown back on to the wrapping fallback policies will be an HttpRequestException
, not BrokenCircuitException
. 因此,如果您的电路在重试耗尽时已恢复为半开状态,则返回到包装回退策略的异常将是
HttpRequestException
,而不是BrokenCircuitException
。
.Handle<Exception>(e => (e is HttpRequestException || e.InnerException is HttpRequestException))
clauses may catch a CircuitBreakerException
s having InnerException is HttpRequestException
.Handle<Exception>(e => (e is HttpRequestException || e.InnerException is HttpRequestException))
子句可能会捕获CircuitBreakerException
,而InnerException is HttpRequestException
A CircuitBreakerException
contains the exception which caused the circuit to break as its InnerException
. CircuitBreakerException
包含导致电路中断的异常,作为其InnerException
。 So an overly greedy/looser check of e.InnerException is HttpRequestException
could also catch a CircuitBreakerException
having InnerException is HttpRequestException
. 因此,对
e.InnerException is HttpRequestException
的过度贪婪/较松懈的检查也可能捕获到具有InnerException is HttpRequestException
的CircuitBreakerException
。 This may or may not be desired depending on your goal. 这可能取决于您的目标,也可能并非如此。
I believe this not to be happening for the original posted code because of the particular way it is constructed. 我相信对于原始发布的代码,由于其特殊的构造方式,这种情况不会发生。 Blocking on the
Task
returned by HttpClient.DoSomethingAsync(...)
causes already an AggregateException->HttpRequestException
, meaning a resulting CircuitBreakerException
nests the HttpRequestException
two-deep: 阻塞
HttpClient.DoSomethingAsync(...)
返回的Task
已导致AggregateException->HttpRequestException
,这意味着生成的CircuitBreakerException
将HttpRequestException
嵌套在两个深度以下:
CircuitBreakerException -> AggregateException -> HttpRequestException
So this doesn't fall in to the one -deep check in the posted code. 因此,这不属于已发布代码中的一次深度检查。 However, be aware that a
CircuitBreakerException
contains the exception which caused the circuit to break as its InnerException
. 但是,请注意
CircuitBreakerException
包含导致电路中断的异常,作为其InnerException
。 This could cause a handle clause checking only e.InnerException is HttpRequestException
to unwantedly (unexpectedly, if it is not your goal) retry for a CircuitBreakerException
, if either: 这可能会导致仅检查
e.InnerException is HttpRequestException
的handle子句不必要地 (如果不是您的目标,则是意外的)重试CircuitBreakerException
,如果存在以下情况:
(a) the code is changed to async
/ await
, which will remove the AggregateException
and so cause the nesting to be only one-deep (a)将代码更改为
async
/ await
,这将删除AggregateException
,因此导致嵌套仅深一层
(b) the code is changed to Polly's .HandleInner<HttpRequestException>()
syntax , which is recursively-greedy, and so would catch a nested-two-deep CircuitBreakerException->AggregateException->HttpRequestException
. (b)将代码更改为Polly的
.HandleInner<HttpRequestException>()
语法 ,该语法是递归贪婪的,因此将捕获嵌套的两个深度的CircuitBreakerException->AggregateException->HttpRequestException
。
Suggestions /* commented out */ // with additional explanation
in the above code suggest how the code-as-posted may be adjusted so that fallbackForCircuitBreaker
invokes as expected. 建议
/* commented out */ // with additional explanation
在上面的代码中/* commented out */ // with additional explanation
,建议如何调整所发布的代码,以便fallbackForCircuitBreaker
按预期调用。
Two other thoughts: 另外两个想法:
async
/ await
throughout if possible. async
/ await
。 Blocking on HttpClient.DoSomethingAsync()
by calling .Result
can impact performance, or risk deadlocks if mixed with other async code, and is bringing in the whole AggregateException
-with- InnerException
pain. 通过调用
.Result
在HttpClient.DoSomethingAsync()
进行阻塞可能会影响性能,如果与其他异步代码混合会导致死锁,并带来整个AggregateException
-with- InnerException
痛苦。
HttpClient
instances. HttpClient
实例的处置和生存期。 (Kept these points 3 and 4 intentionally brief as widely discussed elsewhere.) (保留了第3点和第4点的内容,这在其他地方已广泛讨论过。)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.