ASP.NET 核心 ILogger Memory Windows 10 (1607) / ZAEA2342289CE3AA9B340Z6 服务器泄漏

Question

TL;DR

I'm pretty sure now that there is a memory leak in the conhost process in Windows 10 (version 1607) and Windows Server 2016 that hasn't been there in Windows Server 2012 R2 and is fixed in Windows 10 (version 1809+) and Windows Server 2019.

I couldn't find any official reference to that bug, if anyone can point that out for me I'd appreciate it.

Long story

I'm currently investigating a memory-leak in our (self-hosted) ASP.NET Core Application that results in a high memory usage not of the.exe itself but of the corresponding 'conhost' process. That will only happen on all production servers / VMs, running the same application my laptop (or any dev laptop) does not produce a memory leak.

Dev-Machine (running now for > 2h):

Production Server (running for ~ 10min):

That memory will pile up indefinitely until the server runs out of memory.

That is the code I'm reproducing it with (again: on our production systems, not on dev machines). In our real application it is actually EFCore logging causing the issue (EFCore also uses the ASP.NET Core Loggers):

public class LoggingBackgroundService : BackgroundService
{
    private readonly ILogger<LoggingBackgroundService> logger;

    public LoggingBackgroundService(ILogger<LoggingBackgroundService> logger)
    {
        this.logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        for (var i = 0; i < 10; i++)
        {
            Task.Run(() => RunBackgroundThread(stoppingToken));
        }

        await Task.Delay(Timeout.Infinite, stoppingToken);
    }

    private async Task RunBackgroundThread(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            for (var i = 0; i < 1000; i++)
            {
                logger.LogInformation(GetLogMessage());
                //Console.WriteLine(GetLogMessage());
            }

            await Task.Delay(1000);
        }
    }

    private string GetLogMessage()
    {
        return DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss.ffff");
    }
}

Using Console.WriteLine() instead of the logger works and will not produce a memory leak. And again one more time, that code works fine on a dev machine.

We are using .NET Core 3.1. Can anyone explain what's going on here?

EDIT: My test-system currently sits at ~9 GB RAM, so maybe there is a limit but that's too much RAM spent on a 'conhost' process in any case.

I also tried to use DebugDiag to analyze a memory dump, but that shows a stack trace that doesn't really help me (what else to expect though?):

LeakTrack+13277       
ConhostV2!MergeAttrStrings+b8       
ConhostV2!WriteCharsLegacy+a9c       
ConhostV2!DoSrvWriteConsole+4ea       
ConhostV2!ConsoleIoThread+36b       
kernel32!BaseThreadInitThunk+14       
ntdll!RtlUserThreadStart+21 

I only have a basic understanding of VMs but currently it looks like it is only happening in VMWare ESX (?) systems? I no longer think VMWare is causing the issue but was misled by the different Windows 10 Versions / Builds.

• [Leak] Windows Server 2016 (on VMWare ESX)
• [Leak] Windows 10, 1607 (on VMWare ESX)
• [Leak] Windows 10, 1607 (on Azure)
• [Leak] Windows Server 2016 (on Azure)
• [Leak] Windows Server 2016 (on AWS)
• [Leak] Windows 10, 1607 (on Hyper-V)
• [Leak] Windows Server 2016 (on Hyper-V)
• [No Leak] Windows 10, 1909 (on VMWare Workstation)
• [No Leak] Windows Server 2019 (on Azure)
• [No Leak] Windows 10, 1909 (on Hyper-V)
• [No Leak] Windows Server 2019 (on Hyper-V)
• [No Leak] Windows 10, 1809 (on hardware)

Answer 1

I reported this bug to Microsoft and it has since been fixed.

... with the May 2021 Windows Server 2016 Update :

A fix has been included in the May 2021 Windows Server 2016 update (KB5003197) but is hidden behind a feature flag. To activate it

1. Install KB5003197
2. Add this registry entry: reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides /v 1402424459 /t REG_DWORD /d 1 /f
3. Restart the system

... with the June 2021 Windows Server Update:

This fix should be active with the June-Update without using the registry entry but I haven't tested this.