Java Netty: memory leak problem with DefaultChannelHandlerContext

Question

Hi I have a memory leakage problem and i can't find any solution. My server code:

 bootstrap.channel(NioServerSocketChannel.class)
            .childHandler(new ChannelInitializer<SocketChannel>() {
                @Override
                protected void initChannel(SocketChannel socketChannel) throws Exception {
                    SSLEngine engine = engineCreator.createSSLEngine(credentials);
                    if (engine == null) {
                        throw new Exception("Can't create SSLEngine");
                    }
                    socketChannel.pipeline().addFirst("ssl", new SslHandler(engine));

                    socketChannel.pipeline().addLast("Object-decoder", new ObjectDecoder(
                            ClassResolvers.softCachingConcurrentResolver(
                                    Message.class.getClassLoader())));
                    socketChannel.pipeline().addLast("Object-encoder", new ObjectEncoder());

                //another pair of handles
                }
            })
            .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
            .childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);

code for EngineCreator:

 private static SSLEngine sslEngine = null;

public SSLEngine createSSLEngine(Credentials credentials) {
    try {
        if(sslEngine == null){
            sslEngine = getSSLContext(credentials).createSSLEngine();
            sslEngine.setUseClientMode(false);
            sslEngine.setNeedClientAuth(true);
        }

        return sslEngine;
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

private SSLContext getSSLContext(Credentials credentials) throws Exception {
    InputStream keyStoreInputStream = new FileInputStream(credentials.getKeyStorePath());
    KeyStore keyStore = KeyStore.getInstance("JKS");
    keyStore.load(keyStoreInputStream, credentials.getKeyStorePassword().toCharArray());
    keyStoreInputStream.close();

    KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("SunX509");
    keyManagerFactory.init(keyStore, credentials.getKeyStorePassword().toCharArray());

    TrustManager[] trustManagers = new TrustManager[]{
            new ReloadableX509TrustManager(credentials)
    };
    SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
    sslContext.init(keyManagerFactory.getKeyManagers(), trustManagers, null);
    return sslContext;
}

class ReloadableX509TrustManager implements X509TrustManager {
    private final String trustStorePath;
    private X509TrustManager trustManager;
    private long listTimeModTrust;
    private final Credentials credentials;

    public ReloadableX509TrustManager(Credentials credentials) throws Exception {
        this.credentials = credentials;
        this.trustStorePath = credentials.getTrustStorePath();
        reloadTrustManager(credentials);
    }

    @Override
    public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws 
        CertificateException {
        if (checkTimeTruststore()) {
            try {
                reloadTrustManager(credentials);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        String username = x509Certificates[0].getSubjectDN().getName().split("CN=")[1].split(",")[0];
        x509Certificates[0].checkValidity();
        trustManager.checkClientTrusted(x509Certificates, s);
        x509Certificates[0].checkValidity();
    }


    @Override
    public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws 
         CertificateException {
      //            trustManager.checkServerTrusted(x509Certificates, s);
    }

    @Override
    public X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[0];
    }

    private void reloadTrustManager(Credentials credentials) throws Exception {
        listTimeModTrust = new File(trustStorePath).lastModified();
        KeyStore trustStore = KeyStore.getInstance("JKS");
        InputStream in = new FileInputStream(trustStorePath);
        try {
            trustStore.load(in, credentials.getTrustStorePassword().toCharArray());
        } finally {
            in.close();
        }

        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance("SunX509");
        trustManagerFactory.init(trustStore);

        TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
        for (int i = 0; i < trustManagers.length; i++) {
            if (trustManagers[i] instanceof X509TrustManager) {
                trustManager = (X509TrustManager) trustManagers[i];
                return;
            }
        }
        throw new NoSuchAlgorithmException("No X509TrustManager in TrustManagerFactory");
    }

    private boolean checkTimeTruststore() {
        boolean isNewVersion = false;
        File file = new File(trustStorePath);
        Long lastTime = file.lastModified();
        if (lastTime != listTimeModTrust) {
            isNewVersion = true;
        }
        return isNewVersion;
    }

}

My truststore have 2k ceritificate inside. Heap dump: screen from eclipse memory leak plugin

Currently, after a couple of days of server operation, OutOfMemoryException occurs and the server is down. Does anyone know how to fix it?

Answer 1

I suspect the problem may be that you create a new SslContext object per connection. You should create it once and then share it.

Answer 2

Have you considered running the memory leak detector? https://netty.io/wiki/reference-counted-objects.html

If the issue isn't the sslContext, I suspect it might be in ObjectDecoder or ObjectEncoder if you are not releasing the ByteBuf when you are done with it.

Answer 3

I also encountered a similar problem, after a couple of days of server operation, OutOfMemoryException occurs and the server is down. The official statement is that the Jdk version is a problem https://github.com/reactor/reactor-netty/issues/1443 . My JDK version has been upgraded to U302, but this problem still exists.