memory full and GC take very long time

Question

1. The application GC take a long time and without response when an upstream interface become slow. The thread were BLOCKED when I use jstack pid.

java -Xmx4g -Xms4g -Xmn1g -XX:MetaspaceSize=256M -XX:+PrintGCDateStamps -Xloggc:/tmp/http_gc.log -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC

2019-03-26T22:58:15.846+0800: 25665.934: [GC (Allocation Failure) [PSYoungGen: 4175385K->14818K(4179456K)] 4411887K->541974K(8373760K), 19.0466380 secs] [Times: user=149.46 sys=0.56, real=19.05 secs]
2019-03-26T22:58:37.604+0800: 25687.692: [GC (Allocation Failure) [PSYoungGen: 4179426K->14832K(3154432K)] 4706582K->1684012K(7348736K), 68.8601905 secs] [Times: user=537.37 sys=4.03, real=68.86 secs]
2019-03-26T22:59:48.585+0800: 25758.673: [GC (Allocation Failure) [PSYoungGen: 3154416K->527352K(3666944K)] 4823596K->2421091K(7861248K), 47.9907468 secs] [Times: user=373.32 sys=2.99, real=47.99 secs]
2019-03-26T23:00:38.452+0800: 25808.540: [GC (Allocation Failure) [PSYoungGen: 3666936K->527346K(2592768K)] 5560675K->3195781K(6787072K), 62.5120261 secs] [Times: user=485.91 sys=5.77, real=62.51 secs]
2019-03-26T23:01:42.212+0800: 25872.300: [GC (Allocation Failure) [PSYoungGen: 2592754K->424209K(3129856K)] 5261189K->3619241K(7324160K), 53.6535209 secs] [Times: user=417.13 sys=4.36, real=53.65 secs]
2019-03-26T23:02:37.104+0800: 25927.192: [GC (Allocation Failure) [PSYoungGen: 2489617K->417182K(3204096K)] 5684649K->4033835K(7398400K), 50.0956018 secs] [Times: user=389.61 sys=4.15, real=50.09 secs]
2019-03-26T23:03:27.200+0800: 25977.288: [Full GC (Ergonomics) [PSYoungGen: 417182K->0K(3204096K)] [ParOldGen: 3616653K->3881724K(4194304K)] 4033835K->3881724K(7398400K), [Metaspace: 109373K->108606K(1153024K)], 81.2032824 secs] [Times: user=620.26 sys=16.53, real=81.21 secs]
2019-03-26T23:04:49.719+0800: 26059.807: [Full GC (Ergonomics) [PSYoungGen: 2176000K->95467K(3204096K)] [ParOldGen: 3881724K->4194180K(4194304K)] 6057724K->4289648K(7398400K), [Metaspace: 108606K->108598K(1153024K)], 73.8241548 secs] [Times: user=562.17 sys=16.43, real=73.83 secs]
2019-03-26T23:06:04.738+0800: 26134.826: [Full GC (Ergonomics) [PSYoungGen: 2176000K->521564K(3204096K)] [ParOldGen: 4194180K->4194270K(4194304K)] 6370180K->4715835K(7398400K), [Metaspace: 108598K->108594K(1153024K)], 121.4262195 secs] [Times: user=925.74 sys=25.90, real=121.43 secs]
2019-03-26T23:08:06.936+0800: 26257.024: [Full GC (Ergonomics)

2. We've to use a hystrix circuit breaker with the upstream interface, as well as callable for our own interface.

    @HystrixCommand(fallbackMethod = "fallBack",
            threadPoolProperties = {
                    @HystrixProperty(name = "coreSize", value = "20"),
                    @HystrixProperty(name = "maxQueueSize", value = "50"),
                    @HystrixProperty(name = "queueSizeRejectionThreshold", value = "20")},
            commandProperties = {
                    //使用线程隔离时，调用的超时时间，默认1s
                    @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1000"),
                    @HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "5000"),
                    //出错率超过30%时开始熔断,默认50%
                    @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "30"),
                    @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "10")
            })


        ResponseCallable callable = new ResponseCallable() {
            @Override
            public Response call() throws Exception {
                Response response = new Response();
                response.setData(commonService.getInfo(data));
                response.setSuccess();
                return response;

The problem memory info is below:

  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00 100.00 100.00  92.82  89.17    122  306.980     4  276.454  583.434

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
1028096.0 990208.0  0.0    0.0   2176000.0 2176000.0 4194304.0  4193820.7  116992.0 108641.8 13312.0 11870.2    122  306.980  13   1515.771 1822.751
1028096.0 990208.0  0.0    0.0   2176000.0 2176000.0 4194304.0  4193820.7  116992.0 108641.8 13312.0 11870.2    122  306.980  13   1515.771 1822.751

Below is the jmap instance:

 num     #instances         #bytes  class name
----------------------------------------------
   1:     137721956     3305326944  java.util.LinkedList$Node
   2:     137703668     2203258688  com.dangdang.ddframe.rdb.sharding.parsing.parser.expression.SQLIgnoreExpression

I can't attach the progress to get the thread info:

jstack 1095
1095: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Server compiler detected.
JVM version is 25.65-b01

using thread-local object allocation.
Parallel GC with 38 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 8589934592 (8192.0MB)
   NewSize                  = 4294967296 (4096.0MB)
   MaxNewSize               = 4294967296 (4096.0MB)
   OldSize                  = 4294967296 (4096.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 536870912 (512.0MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 2228224000 (2125.0MB)
   used     = 2228201192 (2124.9782485961914MB)
   free     = 22808 (0.02175140380859375MB)
   99.99897640452666% used
From Space:
   capacity = 1052770304 (1004.0MB)
   used     = 0 (0.0MB)
   free     = 1052770304 (1004.0MB)
   0.0% used
To Space:
   capacity = 1013972992 (967.0MB)
   used     = 0 (0.0MB)
   free     = 1013972992 (967.0MB)
   0.0% used
PS Old Generation
   capacity = 4294967296 (4096.0MB)
   used     = 4294824496 (4095.863815307617MB)
   free     = 142800 (0.1361846923828125MB)
   99.99667517840862% used

Answer 1

firstly you have to check for your bad codes.

secondly you can use a garbageCollector invoker like this

@Component
public class GarbageCollectorInvoker {
    private final Logger logger = LoggerFactory.getLogger(GarbageCollectorInvoker.class);

    @Scheduled(cron = "0 0/30 * * * *")
    public void execute() {
        logger.warn("============GC Status B4=================");
        logger.warn("Heap size free (mb): " + Runtime.getRuntime().freeMemory() / 1024 / 1024);
        logger.warn("Heap size total (mb): " + Runtime.getRuntime().totalMemory() / 1024 / 1024);
        logger.warn("Heap size max (mb): " + Runtime.getRuntime().maxMemory() / 1024 / 1024);

        System.gc();

        logger.warn("============GC Status After=================");
        logger.warn("Heap size free (mb): " + Runtime.getRuntime().freeMemory() / 1024 / 1024);
    }
}