Spring Boot（2.0.3.RELEASE）应用程序中的内存泄漏

Question

We are having a memory leak in a high load(6m req/day) Spring boot application, serving few rest endpoints. old_gen grow slowly over time and committed memory reaches the container limit, causing container showdown. After taking several heapdump, I was able to find out the following:

1. A good amount of memory is used by org.apache.catalina.webresources.CachedResource, with an instance count of around 12000, causing similar amount of other objects including java.io.File. I tried disabling tomcat caching, while these classes are gone from heapdump but the throughput decreases inevitably. Any suggestions on the best practice?

2. Spring actuator also consumes good amount of memory

 Class Name | Objects | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------------- org.springframework.boot.actuate.audit.InMemoryAuditEventRepository | 1 | 24 | >= 5,670,928 org.springframework.boot.actuate.audit.AuditEvent[] | 1 | 4,016 | >= 5,670,880 org.springframework.boot.actuate.audit.AuditEvent | 1,000 | 32,000 | >= 5,666,416 org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationDetails| 1,000 | 40,000 | >= 5,319,552 -------------------------------------------------------------------------------------------------------------------------------- 

It seems to be stable after reaching this level. But in that case on every request, the objects are being replaced in the array. The code(InMemoryAuditEventRepository.java) looks like below, not sure - maybe this causes some leak.

this.events[this.tail] = event;

Any suggestions on this? disable spring actuator?

3. 17670 instances of this class: org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey. Also seems stable like previous class. Seems like related to caching.
4. 20000 instances of com.mysql.jdbc.ConnectionPropertiesImpl$BooleanConnectionProperty!

Path to GC shows:

 <pre> Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller @ 0xcadf60f8 NioBlockingSelector.BlockPoller-1 Thread| 16,884 | 136 | 1,080,576 | 920 '- contextClassLoader org.springframework.boot.loader.LaunchedURLClassLoader @ 0xc8018578 | 16,884 | 80 | 1,080,576 | 4,333,096 '- classes java.util.Vector @ 0xc82b5c90 | 16,884 | 32 | 1,080,576 | 2,706,712 '- elementData java.lang.Object[20480] @ 0xca1a5dd0 | 16,884 | 81,936 | 1,080,576 | 2,706,680 '- [7996] class com.mysql.jdbc.NonRegisteringDriver @ 0xc9b94bd8 | 16,884 | 112 | 1,080,576 | 7,272 '- connectionPhantomRefs java.util.concurrent.ConcurrentHashMap @ 0xc9b94ed8 | 16,884 | 64 | 1,080,576 | 6,528 '- table java.util.concurrent.ConcurrentHashMap$Node[256] @ 0xcc905b78 | 16,884 | 1,040 | 1,080,576 | 6,464 |- [15] java.util.concurrent.ConcurrentHashMap$Node @ 0xcc963078 | 504 | 32 | 32,256 | 128 | '- val, key com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference @ 0xcbfa3840 | 504 | 32 | 32,256 | 48,504 | |- discovered com.mysql.jdbc.NonRegisteringDriver$ConnectionPhantomReference @ 0xcbb4ec50| 378 | 32 | 24,192 | 48,504 | |- referent com.mysql.jdbc.JDBC4Connection @ 0xcc080018 | 126 | 1,232 | 8,064 | 32,024 | '- Total: 2 entries | | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 

Any clue?

5. Heapdump also shows some unreachable objects. Count 128,249 consuming 7MB, major of which are:

 <pre> Class Name | Objects | Shallow Heap --------------------------------------------------------------------------- char[] | 21,936 | 1,988,712 int[] | 13,191 | 1,913,760 java.lang.Object[] | 12,230 | 652,176 java.lang.String | 21,936 | 526,464 java.lang.reflect.Method | 2,994 | 263,472 java.lang.ref.SoftReference | 4,051 | 162,040 java.util.LinkedHashMap$Entry | 3,873 | 154,920 java.util.HashMap$Node[] | 2,470 | 142,784 java.util.LinkedHashMap | 2,314 | 129,584 java.lang.reflect.Constructor | 1,133 | 90,640 sun.reflect.generics.tree.SimpleClassTypeSignature| 2,776 | 66,624 java.util.ArrayList | 2,735 | 65,640 --------------------------------------------------------------------------- 

Can anyone suggests how to see the value of these objects (using MAT), how to track down?

Any other suggestions?

Answer 1

After analyzing the heapdumps, its not quite obvious if there is actually any memory leak or it is JVM's usual behavior. We were using docker image with oracle jre 8, which was last updated 2 years ago. We changed to openjdk and now the memory behavior seems much more stable.