BackgroundService class running in Docker linux container does not shutdown gracefully

Question

I've created a worker service that inherits from Microsoft.Extensions.Hosting.BackgroundService which I then deploy to a docker linux container on my windows machine through visual studio debugger. I put a breakpoint on code that happens when the cancellationtoken.IsCancellationRequested is true. I then issue a "docker stop --time=30" to the container, the breakpoint is never hit and after 30 seconds the debugger stops forcefully.

I also tried overriding the StopAsync method and put a breakpoint in there and that also does not get called. I am running .net core 3, latest version of docker desktop. I have confirmed that StartAsync gets called.

This is my program file.

 public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }


        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((hostingContext, config) =>
                {
                    var builder = new ConfigurationBuilder().SetBasePath(Directory.GetCurrentDirectory())
                        .AddEnvironmentVariables().Build();
                })
                .ConfigureServices((hostContext, services) => { services.AddHostedService<Worker>();    });

    }

If anyone has an idea of what I missed, or a working example for a non webserver service that respects the stop i'd be very grateful.

Adding what my worker looks like:

using System;
using System.Threading;
using System.Threading.Tasks;

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace MyWorker
{
    public class Worker : BackgroundService
    {    
        public Worker(ILogger<Worker> logger, IConfiguration configs)
        {

        }

        public override Task StopAsync(CancellationToken cancellationToken)
        {
           return base.StopAsync(cancellationToken);   
        }

        public override Task StartAsync(CancellationToken cancellationToken)
        { 
            return base.StartAsync(cancellationToken);
        }


        protected override async Task ExecuteAsync(CancellationToken cancellationToken)
        {
            try
            {
                await DoWork(cancellationToken); //While loop that checks token in here
            }

            catch (Exception ex)
            {

                throw;
            }
        }
    }
}

Answer 1

To listen to Ctrl+C, SIGINT and SIGTERM you need to add Console Lifetime support , either through UseConsoleLifetime() or RunConsoleAsync ,eg:

public static async Task Main(string[] args)
{
    CreateHostBuilder(args).Build().RunConsoleAsync();
}

or

public static void Main(string[] args)
{
    CreateHostBuilder(args).UseConsoleLifeTime().Build().Run();
}

If shutting down takes longer than the timeout set in ShutdownTimeout , a warning is logged.

How it works

Knowing how the Console lifetime works can help troubleshoot delays.

If you check the source, RunConsoleAsync is nothing more than a call to UseConsoleLifeTime().Build().RunAsync();

The internal ConsoleLifetime class adds event listeners for the Cancel and ProcessExit events:

AppDomain.CurrentDomain.ProcessExit += OnProcessExit;
Console.CancelKeyPress += OnCancelKeyPress;

Ctrl+C will only forwart the Stop request to the host :

private void OnCancelKeyPress(object sender, ConsoleCancelEventArgs e)
{
    e.Cancel = true;
    ApplicationLifetime.StopApplication();
}

The ProcessExit handler on the other hand will emit a warning if shutting down takes longer than the timeout specified in HostOptions.ShutdownTimeout :

private void OnProcessExit(object sender, EventArgs e)
{
    ApplicationLifetime.StopApplication();
    if(!_shutdownBlock.WaitOne(HostOptions.ShutdownTimeout))
    {
        Logger.LogInformation("Waiting for the host to be disposed. Ensure all 'IHost' instances are wrapped in 'using' blocks.");
    }
    _shutdownBlock.WaitOne();
    // On Linux if the shutdown is triggered by SIGTERM then that's signaled with the 143 exit code.
    // Suppress that since we shut down gracefully. https://github.com/aspnet/AspNetCore/issues/6526
    System.Environment.ExitCode = 0;
}

Answer 2

Well after 4 days I finally figured it out, the problem is not with the code or the docker, the problem is with the Visual Studio debugger. If you are running your code by pressing play in VS2019 it will not respect the SIGTERM. If you run through the manual docker run command you will see your code respond to a DOCKER STOP command.

There might be a way to get it to work eventually but I haven't found it.

Answer 3

Encrease shutdown timeout work for me. Thank you Panagiotis for share MS Docs.

Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
    services.Configure<HostOptions>(option =>
    {
        option.ShutdownTimeout = System.TimeSpan.FromSeconds(20);
    });
});