Memory leak/issue of the object io.netty.buffer.PooledByteBufAllocator from JAR netty-buffer transitively used by azure-data-appconfiguration

Question

About application.

We are running a Kafka streaming application in Azure Kubernetes Service (AKS). Which consumes message from Azure Event hubs, performs business logic and writes output message to another event hub. We store our configurations in Azure App Configuration Service and read all configs at once and cache it in an object and reuse it when required.

Jar azure-data-appconfiguration:jar:1.2.4 is used to connect to App Configuration, which transitively download the jar io.netty:netty-buffer:jar:4.1.68.Final:compile. Below is the dependency tree

+- com.azure:azure-data-appconfiguration:jar:1.2.4:compile
|  +- com.azure:azure-core:jar:1.22.0:compile
|  \- com.azure:azure-core-http-netty:jar:1.11.2:compile
|     +- io.netty:netty-buffer:jar:4.1.68.Final:compile

Please note: As part of businees logic we apply Changed file to Base file to generate new files. We use class "https://github.com/google/diff-match-patch/blob/master/java/src/name/fraser/neil/plaintext/diff_match_patch.java"

ISSUE

The issue we are facing is memory utilization is increasing gradually for this streaming application. When We took heap dump for analysis We found that object io.netty.buffer.PooledByteBufAllocator is occupying more space we suspect leak here. We can not exclude netty-buffer because it is used to connect to Azure App Configuration.

What We tried we upgraded the jar azure-data-appconfiguration to latest version 1.3.6 thinking may be there may be memory leak in older version, But still same issue persist. below is the dependency tree.

+- com.azure:azure-data-appconfiguration:jar:1.3.6:compile
|  +- com.azure:azure-core:jar:1.31.0:compile
|  \- com.azure:azure-core-http-netty:jar:1.12.4:compile
|     +- io.netty:netty-buffer:jar:4.1.79.Final:compile

Support required

1. Is there any known memory leak in netty-buffer jar?
2. Is there any way to know stack trace where these objects are getting created.
3. Please suggest best way to debug this leak and fix.

Answer 1

This seems to be a possible issue with the SDK. An issue has been created on the GitHub repo to track see: https://github.com/Azure/azure-sdk-for-java/issues/30666

Answer 2

I've created a simple AppConfiguration based application which created 500 keys, listed them, and then deleted them while taking periodic heap dumps and I didn't find any leaks. The mentioned GitHub issue ( https://github.com/Azure/azure-sdk-for-java/issues/30666 ) was for a code path that an AppConfiguration client won't use.

Based on the heap dump image posted this looks to be normal memory behavior. PooledByteBufAllocator will allocate 2 "arenas" per thread and each arena will allocate a 4MB chunk, which if the machine being used is 2 threads then that's the 16MB highlighted. See this PR for more information: https://github.com/netty/netty/pull/12108