Kafka SSL connectivity issue with application on IBM Websphere Application server

Question

I am working on integrating my application with Apache Kafka. While everything works as expected while connecting to test brokers.
I have hit a blocker with 2-way SSL in preprod environment. My application is deployed in Websphere Application server and the certs/keys are maintained in a Websphere keyring. Issue is Kafka producer configuration is unable to interact with the keyring to find the trusted certs or the key and hence the connectivity is failing.
I can't use JKS files since that will defeat the purpose of the keyring and is against the application design. The whole problem appears to be the interaction of Kafka client code with keyring, during application startup. Any suggestion on this is appreciated.

org.apache.kafka.common.network.Selector) - [Producer clientId= xxxxxxx] Connection with disconnected due to authentication exception
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at com.ibm.jsse2.bb.B(bb.java:525)
        at com.ibm.jsse2.oc.b(oc.java:394)
        at com.ibm.jsse2.oc.c(oc.java:146)
        at com.ibm.jsse2.oc.wrap(oc.java:316)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:39)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:434)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:299)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:253)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:79)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:486)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:424)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
        at java.lang.Thread.run(Thread.java:798)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at com.ibm.jsse2.k.a(k.java:5)
        at com.ibm.jsse2.oc.a(oc.java:170)
        at com.ibm.jsse2.bb.a(bb.java:560)
        at com.ibm.jsse2.bb.a(bb.java:432)
        at com.ibm.jsse2.cb.a(cb.java:30)
        at com.ibm.jsse2.cb.a(cb.java:394)
        at com.ibm.jsse2.bb.t(bb.java:170)
        at com.ibm.jsse2.bb$1.a(bb$1.java:4)
        at com.ibm.jsse2.bb$1.run(bb$1.java:2)
        at java.security.AccessController.doPrivileged(AccessController.java:492)
        at com.ibm.jsse2.bb$c_.run(bb$c_.java:11)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:388)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:468)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:326)
        ... 8 more
Caused by: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
        java.security.cert.CertPathValidatorException: The certificate issued by xxxxxxxxxx is not trusted; internal cause is: 
        java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.jsse2.util.f.a(f.java:70)
        at com.ibm.jsse2.util.f.b(f.java:95)
        at com.ibm.jsse2.util.e.a(e.java:20)
        at com.ibm.jsse2.zc.a(zc.java:35)
        at com.ibm.jsse2.zc.a(zc.java:156)
        at com.ibm.jsse2.zc.checkServerTrusted(zc.java:125)
        at com.ibm.jsse2.cb.a(cb.java:302)
        ... 17 more
Caused by: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
        java.security.cert.CertPathValidatorException: The certificate issued by xxxxxxxxxxx is not trusted; internal cause is: 
        java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:410)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:256)
        at com.ibm.jsse2.util.f.a(f.java:144)
        ... 23 more
Caused by: java.security.cert.CertPathValidatorException: The certificate issued by xxxxxxxxxxxxxxx is not trusted; internal cause is: 
        java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.security.cert.BasicChecker.<init>(BasicChecker.java:111)
        at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:176)
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.myValidator(PKIXCertPathBuilderImpl.java:737)
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:649)
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.buildCertPath(PKIXCertPathBuilderImpl.java:595)
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:356)
        ... 25 more
Caused by: java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.security.cert.CertPathUtil.findIssuer(CertPathUtil.java:316)
        at com.ibm.security.cert.BasicChecker.<init>(BasicChecker

Answer 1

I'm not sure if this is going to work but you can try this:

1. extract the certificate and key from Keyring before you start up the producer;
2. save them in *.jks files somewhere in your machine;
3. pass the path of your newly created Keystore and Truststore to your Kafka producer

Unfortunately, Java Kafka Clients can only interact with *.jks files so a proper conversion needs to be done before start-up. Another option would be to do the same thing but at a pre-deployment phase ( before starting up your application, you prepare the Keystore and Truststore ).

Answer 2

I know I'm a bit late to the party but I was looking for a solution to this issue as well and found a way to provide a Kafka Producer with WAS SSL configuration data. The idea came from the following IBM documentation that explains how to use the com.ibm.websphere.ssl.JSSEHelper to obtain information specified in an WAS managed SSL Configuration: https://www.ibm.com/docs/en/was/8.5.5?topic=ascdoprse-programmatically-specifying-outbound-ssl-configuration-using-jssehelper-api

The following would work to provide SSL configuration data to your Kafka Producer/Consumer properties:

 com.ibm.websphere.ssl.JSSEHelper jsseHelper = JSSEHelper.getInstance();
 Properties sslProperties = jsseHelper.getProperties("<your_ssl_conf_alias>")
 consumerProperties.put(org.apache.kafka.common.config.SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG,sslProperties.getProperty("com.ibm.ssl.trustStore"));

You can use an existing alias or create a new one in the WAS console under:

SSL certificate and key management > SSL configurations