What is the proper way to investigate memory leaks on .net core linux kubernetes container

Question

I have the .net core app running on linux docker container, and while taking dumps (core 2.2 or 3.0) I can't open them in the PerfView,

taking dumps according to this instruction: https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-dump-instructions.md

PerfView shows this error in the logs:

Creating heap dump C:\temp\dumps\dump\dump-1.gcdump from process dump C:\temp\dumps\dump\dump-1.dmp.
HeapDump Error: Microsoft.Diagnostics.Runtime.ClrDiagnosticsException: Could not load crash dump 'C:\temp\dumps\dump\dump-1.dmp', HRESULT: 0x80070057
   at Microsoft.Diagnostics.Runtime.DbgEngDataReader..ctor(String dumpFile)
   at Microsoft.Diagnostics.Runtime.DataTarget.LoadCrashDump(String fileName)
   at GCHeapDumper.InitializeClrRuntime(String processDumpFile, DataTarget& target, ClrRuntime& runtime)
   at GCHeapDumper.DumpHeapFromProcessDump(String processDumpFile)
   at Program.MainWorker(String[] args)

Answer 1

The dump file is created inside the container and therefore not directly accessible from your machine. (If you are running Windows and Docker for Windows there is even a virtual machine inbetween.)

What you need to do is to copy the dumb file from the container to your host and open it afterwards. This can be achieved using docker cp command, for example: docker cp <container name>:<path in container>dump-1.gcdump C:\temp\dumps\dump\dump-1.gcdump

Answer 2

I believe you need to use Linux debugger to open Linux dumps. Afaik PerfView supports only Windows dump.

CoreClr team provides SOS debugger extension that can be utilized from lldb debugger. https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md

Answer 3

Dumps on .NET core are not cross-platform compatible due to cross-platform DAC (For more info, see https://github.com/dotnet/runtime/blob/master/docs/design/coreclr/botr/dac-notes.md ). There have been discussions/plans on supporting this but it hasn't happened yet.

You can use dotnet-gcdump tool and it should be cross-platform compatible. Here is a doc on how to use it: https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-gcdump-instructions.md

A dump taken from dotnet-gcdump can be viewed on PerfView.