为什么在Java 8中使用连续完整GC

Question

I using Netty Bootstrap (v3) for server socket. But after usage time, I got this GC logs:

2018-06-12T10:51:18.219-0400: 331664.849: [Full GC (Ergonomics) [PSYoungGen: 129024K->118028K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852435K->2841439K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 19.7892397 secs] [Times: user=256.57 sys=0.00, real=19.78 secs] 
2018-06-12T10:51:38.014-0400: 331684.645: [Full GC (Ergonomics) [PSYoungGen: 129024K->118091K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852435K->2841503K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 19.6915664 secs] [Times: user=255.44 sys=0.00, real=19.69 secs] 
2018-06-12T10:51:57.716-0400: 331704.347: [Full GC (Ergonomics) [PSYoungGen: 129024K->118166K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852435K->2841577K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 23.3087463 secs] [Times: user=300.78 sys=0.00, real=23.30 secs] 
2018-06-12T10:52:21.036-0400: 331727.667: [Full GC (Ergonomics) [PSYoungGen: 129024K->118197K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852435K->2841608K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 19.5556922 secs] [Times: user=253.78 sys=0.00, real=19.56 secs] 
2018-06-12T10:52:40.598-0400: 331747.229: [Full GC (Ergonomics) [PSYoungGen: 129024K->118252K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852435K->2841664K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 18.9957031 secs] [Times: user=246.53 sys=0.00, real=19.00 secs] 
2018-06-12T10:52:59.601-0400: 331766.232: [Full GC (Ergonomics) [PSYoungGen: 128996K->118259K(144896K)] [ParOldGen: 2723411K->2723411K(2723840K)] 2852408K->2841671K(2868736K), [Metaspace: 17983K->17983K(1064960K)], 19.6780211 secs] [Times: user=255.45 sys=0.00, real=19.68 secs] 

Now, i'm not set any Xms and Xmx parameters in JVMs.

Can anyone analyze it for me? How to optimize this situation? Thanks.

More informations:

I'm using java:

java version "1.8.0_152"

Libraries:

pom.xml

    <dependency>
        <groupId>com.jolbox</groupId>
        <artifactId>bonecp</artifactId>
        <version>${bonecp.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>commons-configuration</groupId>
        <artifactId>commons-configuration</artifactId>
        <version>1.10</version>
    </dependency>

    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty</artifactId>
        <version>3.10.5.Final</version>
    </dependency>

    <dependency>
        <groupId>com.oracle</groupId>
        <artifactId>ojdbc6</artifactId>
        <version>11.2.0.4</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${slf4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>${slf4j.version}</version>
    </dependency>

MessageDecoder

@Override
protected Object decode(ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer) throws Exception {

    Message message = null;
    buffer.markReaderIndex();
    ChannelBuffer rawMessage = buffer.copy();

    try {
        loadConfiguration();

        // skip for message start character (#)
        buffer.skipBytes(1);
        byte[] data = null;
        String messageRef;

        // message reference
        ChannelBuffer msgData = readFrame(buffer, messageRefDelimiter);
        messageRef = new String(msgData.array());

        // get data
        msgData = readFrame(buffer, messageDelimiter);
        data = msgData.array();

        message = new Message();
        message.setMessageType(MessageType.Yyyy);
        message.setMessageReference(Long.parseLong(messageRef));
        message.setData(data);

    } catch (Exception e) {

        String msg = new String(rawMessage.array());

        logger.error("Error when decode message {}", msg, e);
        message = null;
    }

    return message;
}

TaskManager

static {
    try {
        PropertiesConfiguration config = AppConfig.getPropertiesConfiguration();
        int poolSize    = config.getInt(AppEnv.TASK_POOL_SIZE, AppEnv.TASK_POOL_SIZE_DEFAULT_VALUE);
        int runningTask = config.getInt(AppEnv.TASK_RUNNING, AppEnv.TASK_RUNNING_DEFAULT_VALUE);

        worksQueue = new ArrayBlockingQueue<Runnable>(poolSize);
        executor = new ThreadPoolExecutor(runningTask, runningTask * 2, 10, TimeUnit.SECONDS, worksQueue, executionHandler);

        executor.allowCoreThreadTimeOut(true);

        logger.info("Init TaskManager: poolSize={}, runningTask={}", poolSize, runningTask);
    } catch (AppConfigException e) {
        logger.error("Init TaskManager error", e);
    }
}

/**
 * execute message
 * 
 * @param message
 */
public static void execute(Message message) {
    MessageProcessing messageProcessing = new MessageProcessing(message);
    executor.execute(messageProcessing);
}

MessageProcessing

public void processMessage(Message message) {

    DateFormat df = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss");

    String[] data;
    String mes = null;

    try {
        loadConfiguration();
    } catch (Exception e) {
        logger.error("error load configuration", e);
    }

    try {
        mes = new String(message.getData());
        data = mes.split(fieldDelimiter);

        if (data[2].equals("10")) {

            Yyyy10 Yyyy = new Yyyy10();
            Yyyy.message = message.getMessageReference() + "," + mes;
            Yyyy.messageReference = message.getMessageReference();
            Yyyy.imsi = data[0];
            Yyyy.isdn = StringUtil.formatIsdn(data[1]);
            Yyyy.YyyyInformation = Integer.parseInt(data[2]);
            Yyyy.accountProfile = data[3];
            Yyyy.timestamp = df.parse(data[4]);
            Yyyy.refillCount = Integer.parseInt(data[5]);
            Yyyy.mainAmount = Long.parseLong(data[6]);
            Yyyy.bonusAmount = Long.parseLong(data[7]);
            Yyyy.scratchCardNumber = data[8];
            Yyyy.scratchCardProfile = data[9];
            Yyyy.tac = df.parse(data[10] + " 00:00:00");

            //logger.info(Yyyy.toString());

            DbUtil.insertYyyy10(Yyyy);
        }
        else if(data[2].equals("11")) {

            Yyyy11 Yyyy = new Yyyy11();
            Yyyy.message = message.getMessageReference() + "," + mes;
            Yyyy.messageReference = message.getMessageReference();
            Yyyy.imsi = data[0];
            Yyyy.isdn = StringUtil.formatIsdn(data[1]);
            Yyyy.YyyyInformation = Integer.parseInt(data[2]);
            Yyyy.accountProfile = data[3];
            Yyyy.timestamp = df.parse(data[4]);

            //logger.info(Yyyy.toString());

            DbUtil.insertYyyy11(Yyyy);
        } else {
            logger.error("not match Yyyy(10, 11)", message);
        }
    } catch (SQLException e) {

    } catch (Exception e) {
        logger.error("error parse message: {}", message, e);
    } finally {
        try {
            data = null;
            mes = null;
        } catch (Exception e) {}
    }

}

DbUtil

public static void insertYyyy11(Yyyy11 Yyyy) throws SQLException {

    String insertQuery = "INSERT INTO TBL_Yyyy_11(ISDN,MESSAGE_REFERENCE,IMSI,ACCOUNT_PROFILE,TIMESTAMP,MSG_CONTENT) VALUES(?,?,?,?,?,?)";

    try (Connection connection = getConnection(); PreparedStatement preparedStatement = connection.prepareStatement(insertQuery);) {
        preparedStatement.setQueryTimeout(queryTimeout);

        preparedStatement.setString(1, Yyyy.isdn);
        preparedStatement.setLong(2, Yyyy.messageReference);
        preparedStatement.setString(3, Yyyy.imsi);
        preparedStatement.setString(4, Yyyy.accountProfile);
        preparedStatement.setTimestamp(5, DateUtil.toSqlTimestamp(Yyyy.timestamp));
        preparedStatement.setString(6, Yyyy.message);

        preparedStatement.executeUpdate();
    } catch (SQLException e) {
        logger.error("insertYyyy11 error, message: {}", Yyyy, e);
        throw e;
    }
}

Thread dump:

There are a lot of RUNNABLE threads

"pool-5-thread-22" #92 prio=5 os_prio=0 tid=0x00007ff5e0044800 nid=0x55ba runnable [0x00007ff6b30ef000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at oracle.net.ns.Packet.receive(Packet.java:308)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:324)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:268)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:190)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:107)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:350)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1046)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1336)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
    at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3694)
    - locked <0x00000006c6b5b0d0> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1354)
    at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
    at vn.xxx.util.DbUtil.insertYyyy11(DbUtil.java:128)
    at vn.xxx.service.YyyyMessageServiceImpl.processMessage(XXXMessageServiceImpl.java:109)
    at vn.xxx.task.MessageProcessing.run(MessageProcessing.java:33)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

The most likely explanation is that the heap is nearly full. If the heap gets too full, the JVM will run the GC more and more frequently.

Given that the server only started doing that after running for a while, the most likely cause is that you have a memory leak, either in your application, or in some library that you are using.

Assuming my assumptions are correct, then the way to resolve the problem is to use jhat and a memory profiler to try to identify the source of the memory leak.

---

Another couple of possibilities are that you have too many worker threads in your pool (using too much memory) or that your work queue is getting seriously backlogged (using too much memory) or (maybe) that your use of a blocking queue is actually causing backlog earlier in your (asynchronous?) request handling.

If your server cannot keep up with the request rate, then the only viable strategy is to refuse ... or drop ... requests early. Allowing them to backlog will eventually lead to problems.