Named semaphore not working on ASP .Net Core 5 Web API hosted on Azure App Service

Question

I have an ASP.Net Core 5 Web API running on Azure App Service. Once a day, it runs a Hosted BackgroundService. Because I only want this service to run once a day (at 2 AM), and because my App Service has auto scale-out turned on (increase instance by 1 when CPU or Memory > 70%, up to a maximum of 3 instances), I implemented a named semaphore, to prevent more than one instance running simultaneously.

Here is the code for the Hosted BackgroundService. I have simplified it a bit to show the essentials.

public class MyBackgroundService : BackgroundService
{
    private Timer _timer;
    private readonly ILogger<MyBackgroundService> _logger;

    public MyBackgroundService(ILogger<MyBackgroundService> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        TimeSpan timeTillNextRun = CalculateTimeTillNextRun();
        _timer = new Timer(DoWork, null, timeTillNextRun, new TimeSpan(1, 0, 0, 0));
    }

    private async void DoWork(object state)
    {
        try
        {
            _timer.Change(Timeout.Infinite, Timeout.Infinite);
            await RunAsync();
        }

        finally
        {
            TimeSpan timeTillNextRun = CalculateTimeTillNextRun();
            _timer.Change(timeTillNextRun, new TimeSpan(1, 0, 0, 0));
        }
    }

    private async Task RunAsync()
    {
        using (Semaphore semaphore = new Semaphore(1, 1, "MyBackgroundService"))
        {
            if (!semaphore.WaitOne(1))
            {
                return;
            }

            try
            {
                _logger.LogInformation(null, "Started.");                    
                // Do stuff
            }

            finally
            {
                semaphore.Release();
            }
        }
    }
}

So I have a timer which will fire at 2AM each day. When the timer fires, I pause it, and I call RunAsync(). In RunAsunc(), I set a named (system) semaphore. I only wait 1 millisecond because, if the semaphore is in use, I don't even want to wait for it to finish because this background service should run once a day only.

However, when I check the logs, I see it runs up to 6 times simultaneously (well, within about 3 seconds), twice on each service.

id, machine_name, status, time_stamp
'9577', 'RD2818786D33F4', 'Started.', '2021-08-16 02:00:06'
'9578', 'RD2818786D33F4', 'Started.', '2021-08-16 02:00:08'
'9579', 'RD2818786D0367', 'Started.', '2021-08-16 02:00:08'
'9580', 'RD2818786D1D19', 'Started.', '2021-08-16 02:00:09'
'9581', 'RD2818786D0367', 'Started.', '2021-08-16 02:00:09'
'9582', 'RD2818786D1D19', 'Started.', '2021-08-16 02:00:10'

As you can see above, it first ran at 01:00:06, then again 2 seconds later on the same machine, then instantly again on a different machine, then twice more a second later, and finally once more a second after that.

Now, looking at the "machine_name" column, it would appear that each instance of the App Service is running on a different VM. I thought, since all instances are on the same service plan, that it would be the same VM, but I guess I was wrong. So, I guess that's why the semaphore isn't working - it does not work across different machines. The strange thing is that it appears to not be working within the same machine either. As you can see int he log above, RunAsync() ran on each instance twice. Now it's not always exactly like that. For example, this is the log for the previous day:

id, machine_name, status, time_stamp
'9569', 'RD2818786D2D5E', 'Started.', '2021-08-15 02:00:05'
'9570', 'RD2818786D2D5E', 'Started.', '2021-08-15 02:00:06'

Here it only ran twice, on one instance. Again, it should only have run once - especially seeing as it was on the same machine.

Am I using Semaphore wrong? I understand that it probably won't work across VM's, but it doesn't seem to be working even on the same VM.

Any advice would be greatly appreciated. Thanks.

Ps I understand that it would be better for me to remove this hosted service from the API and stick it in an Azure Function or Azure WebJob. And indeed, I will do that ASAP. But I'm still curious to know why my Semaphore is not working.

Answer 1

Semaphore appears to be creating a new semaphore every single time. I don't know why. This is happening both on different processes and even on single processes. I added the "Global//" prefix to the Semaphore name, as suggested by Steeeve, but it made no difference. I even logged the value of the 4th parameter (boolean param) in the Semaphore constructor, again as suggested by Steeeve, to see if it was actually creating a new semaphore each time, and it truly is. I tried multiple things, even removing the Semaphore from the Using() block (because I thought maybe that was destroying the Semaphore in a way it shouldn't) but it made no difference.

Eventually I moved back to a SemaphoreSlim like I had initially. This doesn't give me the option of creating a named/system semaphore, like Semaphore did, but at least it works on single processes.

It appears Semaphore simply doesn't work on an app hosted on Azure App Service, at least one that is set to Auto Scale Out like mine is. I can understand this, as it appears that Azure creates a new VM for each instance of the app when it scales out (at least sometimes - I can't guarantee this is always the case) but the strange thing is that it appears not to be working even on the same VM. From my understanding, a named/system semaphore works at an OS level - ie it's supposed to work across processes within a machine.

Thanks, everyone for your help.