ASP.NET 6 app has a long delay after WebApplication.Run when hosted in IIS

Question

We just converted an ASP.NET app to .NET 6.0 and have it hosted in IIS 10. Since the conversion, deployments become unresponsive for about 5 minutes every time the app is started.

Here is the basic setup:

public static void Main(string[] args)
{
    var builder = WebApplication.CreateBuilder(args);
    builder.Host.ConfigureDefaults(args);
    builder.Services.AddControllersWithViews(ConfigureMvcOptions)
    
    var app = builder.Build();
    Log.Info("Begin Configuration"); 
    app.UseRouting();
    app.MapControllers();
    app.UseStaticFiles();
    app.Lifetime.ApplicationStarted.Register(Started);
    ConfigureSomeOtherIrrelevantSutff();
    Log.Info("Configuration Complete");
    app.Run(); //Delay happens after this
}

private static void Started()
{
    Log.Info($"Application Started");
}

The log indicates the following:

2023-01-17 12:27:50.470: Begin Configuration
2023-01-17 12:28:25.498: Configuration Complete
2023-01-17 12:28:25.580: Application Started
2023-01-17 12:32:57.700: First Request

Note also that IIS AspNetCore Module V2 logs "Application 'C:\IIS\MyApp' started successfully." in Event Viewer

After that, there is a ~4 minute delay before the first request is served. During that time, the web app is entirely unresponsive, the browser just spins indefinitely.

Some things to note that may be relevant:

• We have <handlerSetting name="experimentalEnableShadowCopy" value="true" /> because we were having issues with locked files when deploying. I don't think this is the problem though as the delay is long after the app is copied.
• During the delay, the CPU usage of the IIS worker process is 0%, the app is doing absolutely nothing
• Disable Overlapped Recycle is enabled in IIS, so there is no contention of resources with the old processes -- it is fully terminated by the time the new one starts
• We recycle the app pool after all files are deployed, it doesn't auto-recycle on configuration/file changes

I don't know where to begin to look to diagnose this as it seems to be happening entirely outside of our code. What can I do to further debug the delay?

Edit: Further notes:

• Can't reproduce it locally or on a server that doesn't have load
• I am sometimes experiencing the issue during shutdown as well. Registering for ApplicationStopped shows the app is shut down, then IIS just sits there spinning for X minutes before it starts the new version

Answer 1

I believe I found the issue. It looks like there are outstanding connections that are causing the process to hang during shutdown. I also have overlapped recycle disabled in IIS, so I think the new worker has to wait for the previous one to terminate before starting.

At this point, I don't know why the connections are not being finalized in a more timely manner, so I've just resorted to killing the worker process more quickly by setting the "Shutdown Time Limit" on the AppPool to 10 seconds.