Java Spring Webflux: javax.net.ssl.SSLHandshakeException: Empty client certificate chain

Question

Small question regarding a Spring Webflux application please.

Periodically, but sporadically, in the application logs, I am seeing this stack trace:

2022-02-04 09:20:13.813  WARN [myapplication,,] 11 --- [or-http-epoll-2] .s.ApplicationProtocolNegotiationHandler : [id: 0xc70400fa, L:/.<someIP>:<somePort> ! R:/<someOtherIP>:<someOtherPort>] TLS handshake failed:
javax.net.ssl.SSLHandshakeException: Empty client certificate chain
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[na:na]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336) ~[na:na]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) ~[na:na]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283) ~[na:na]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[na:na]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[na:na]
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[na:na]
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[na:na]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[na:na]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[na:na]
    at java.base/java.security.AccessController.doPrivileged(Native Method) ~[na:na]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[na:na]
    at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.66.Final-linux-x86_64.jar!/:4.1.66.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.66.Final-linux-x86_64.jar!/:4.1.66.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.66.Final-linux-x86_64.jar!/:4.1.66.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.66.Final.jar!/:4.1.66.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.66.Final.jar!/:4.1.66.Final]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

There is unfortunately no pattern as when this appears. On high traffic, on low traffic, even when there is no traffic, this message appears. And this is only seen sometime, not always.

The http client (Spring Webclient) is built as follow:

    @Bean
    @Primary
    public WebClient getWebClient() {
        return WebClient.create().mutate().defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE).clientConnector(new ReactorClientHttpConnector(HttpClient.create().wiretap(true).secure(sslContextSpec -> sslContextSpec.sslContext(getSslContext())))).build();
    }

public SslContext getSslContext() {
        try {
            final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
            try (InputStream file = new FileInputStream(keyStorePath)) {
                final KeyStore keyStore = KeyStore.getInstance(keyStoreType);
                keyStore.load(file, keyStorePassPhrase.toCharArray());
                keyManagerFactory.init(keyStore, keyPassPhrase.toCharArray());
            }

            final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
            try (InputStream trustStoreFile = new FileInputStream(trustStorePath)) {
                final KeyStore trustStore = KeyStore.getInstance(trustStoreType);
                trustStore.load(trustStoreFile, trustStorePassPhrase.toCharArray());
                trustManagerFactory.init(trustStore);
            }

            return SslContextBuilder.forClient().keyManager(keyManagerFactory).trustManager(trustManagerFactory).build();
        } catch (CertificateException | NoSuchAlgorithmException | IOException | KeyStoreException | UnrecoverableKeyException e) {
 LOGGER.error( "Error here: Empty client certificate chain?" + e, e);
            return null;
        }
    }

I am having a hard time understanding the root cause, and the meaning of this stack trace.

Questions:

I see this [id: 0xc70400fa, L:/.someIP:somePort ! R:/someOtherIP:someOtherPort ] What is the meaning of this please? From Spring documentation, L means "local", R means "remote" and. means error, Therefore, does it means the IP in local is trying to "talk to" the IP in remote? but local (me?) presented an empty certificate?

Or is it the opposite remote is trying to talk to me, local, and is presenting an empty certificate?

The request ID (here id: 0xc70400fa) is unusable as there is no way to correlate with anything else. How to properly use this ID please?

Finally, is there a way to get an understanding why and especially when (is it an API call? a health check call, etc) this Empty client certificate chain is happening, and how to prevent it please?

Thank you

Answer 1

There is some config parameters you can check in Spring Boot code to look at their SSL implementation for that kind of thing:

security.require-ssl=true
server.ssl.key-store:/home/user/folder/keystore.p12
server.ssl.key-store-password: XXXXXX              <========== your password
server.ssl.keyStoreType: PKCS12                    <========== your keystore format
server.ssl.keyAlias: XXXXX                         <========== your keystore alias

So you can look at their source code to seek directly their interfaces and provided integrations/implementations.

So it is not required to import a keystore inside our global keystore repository if we do not want it. We can use this another way.

[id: 0xc70400fa, L:/.someIP:somePort: R:/someOtherIP:someOtherPort]

• id: 0xc70400fa => connection id (edited thanks @Violeta)
• L:/ip:port: R:/ip:port => localMachine:localPort did not connect to remoteMachine:remotePort

Or maybe you could try another way to go: using wget, curl or Guzzle or any good HTTP(s) java client... and maybe an Entreprise Service Bus | middleware application like Apache Camel to connect any future webservices or microservices with plug and play way of life.