扩展 SemaphoreSlim 以区分资源使用强度

Question

I have about 100 different external endpoints spread out over about 6 or 7 API's I'm calling from my application. All of these 100 (let's call them) resources share a fair-use pool and collectively keep track of when to give me a 429. I'm limiting my calls with a SemaphoreSlim (found a sweet spot on about 50 parallel calls).

BUT

These resources do not determine rate limiting based on the amount of calls, but on the amount of CPU usage. This means that a call that takes 2ms serverside can be done 1000 times more than a call that takes 2 seconds serverside before triggering 429's.

What I'd like to build is an extension on Semaphore(Slim) that keeps track of avarage roundtrip-times per URL I'm using, store this somewhere and apply limiting based on how "heavy" the calls are that are being queued. So if I want a max of 10 seconds of calls parallel, I can do up to 5000 2ms calls or up to 5 2s calls.

Ideally I'd use a semaphore that allows for blocking more than just one "count". SemaphoreSlim has AwaitAsync and Release. I've implemented it like this:

protected async Task<TResponse> AwaitSemaphoreAndExecuteRequest<TResponse>(Func<Task<TResponse>> ExecutionDelegate)
{
    try
    {
        await _semaphore.WaitAsync(); // Block 1 of 50 
        return await ExecutionDelegate();
    }
    finally
    {
        _semaphore.Release(); // Release 1 of 50
    }
}

But what I'd like to do is something like this:

protected async Task<TResponse> AwaitSemaphoreAndExecuteRequest<TResponse>(Func<Task<TResponse>> ExecutionDelegate, int requestWeight)
{
    try
    {
        await _semaphore.WaitAsync(requestWeight); // Block "weight" of 10000
        return await ExecutionDelegate();
    }
    finally
    {
        _semaphore.Release(requestWeight); // Release "weight" of 10000
    }
}

Does anyone know if there are any best-practices for this, maybe a third party library that already allows for this behaviour? Idea's for implementing something simple?

Answer 1

I created a library that you may want to consider, it's called SemaphoreSlimThrottling that allows you to initialize a SemaphoreSlim with a negative initialCount .

I had created this for a project with a similar problem to yours. I rarely had API issues, but on rare occasions a bunch of calls thrown at the same time caused delays in my API, so what I did was measuring how long the API was taking to answer and if it goes above a certain threshold I would start throttling THEN, and if it went below a certain threshold (a lower value than the previous) then I would stop throttling.

The issue arises when you know you have 25 open calls to the API and you want to start throttling to 20 calls. You can't simply do:

var throttle = new SemaphoreSlim(-5, 20);

because SemaphoreSlims cannot take a negative initial value.

And if you start with an initialCount of 0, you have two problems:

1. if one of those 25 returns and it releases from the semaphore, another can enter, so you effectively are allowing a 25th one in when you intended to only have 20.
2. if 21 return immediately, you can't release 21 if you have a maxCount of 20.

But with SemaphoreSlimThrottling you can simply do:

var throttleMaxCount = 20;
var throttle = new SemaphoreSlimThrottle(currentCallCount - throttleMaxCount, throttleMaxCount);

which basically translates to:

var throttle = new SemaphoreSlimThrottle(-5, 20);

---

If you instead want to take the path of a fully-fledged throttling system, you may want to look at AspNetCoreRateLimit .