Spring Kafka Handshake Failure No X.509 certificate for client authentication, but there is one in my keystore

Question

I have a situation that I can't figure out. Im running in a Springboot app in K8s and trying to consumer from a kafka topic using Spring Kafka within my company and I have to use SSL authentication.

I have mounted my trust & keystores in the pod and when I use keytool I can see the full chain of certificates in there:

Alias name: 4f6b4855-0f06-4202-8531-cf0c8fbc6611
Creation date: Jun 28, 2022
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: C=xx, O=xxxx, CN=xxxxxx
Issuer: C=xx, L=xxx, O=xxxxxxxx, OU=xxxxxx, CN=xxxxxxx
Serial number: 7fbee1f1fb4c0721
Valid from: Thu Jun 23 08:34:51 GMT 2022 until: Fri Jun 23 08:34:51 GMT 2023
Certificate fingerprints:
         SHA1: 6A:27:FF:57:2B:5B:16:97:CA:7C:3B:CE:9D:E3:43:AF:B9:66:EC:AB
         SHA256: AD:96:34:4E:BD:BE:83:B8:5D:09:84:63:55:F0:E6:8A:15:CF:7A:35:6C:36:77:BF:FE:8B:68:A0:BE:61:7E:6B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
......

My properties (yes its a PKCS12 keystore, but a jks extention)

spring.kafka.ssl.trust-store-location=file:/mnt/ssl/truststore.jks
spring.kafka.ssl.trust-store-password=changeit
spring.kafka.ssl.trust-store-type=PKCS12
spring.kafka.ssl.key-store-location=file:/mnt/ssl/keystore.jks
spring.kafka.ssl.key-store-password=changeit
spring.kafka.ssl.key-store-type=PKCS12

When my consumer starts however I get a handshake exception and from the look of it its because it cannot provide the client certificate. Heres the certificate request ( above certificate is signed with the same CA as mentioned in the request)

javax.net.ssl|DEBUG|23|org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1|2022-06-28 16:34:01.595 CEST|CertificateRequest.java:692|Consuming CertificateRequest handshake message (
"CertificateRequest": {
  "certificate types": [ecdsa_sign, rsa_sign, dss_sign]
  "supported signature algorithms": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
  "certificate authorities": [C=xxxx, L=xxxxxx, O=xxxxxxx, OU=Axxxxx, CN=xxxxxx]
}

Yet this the output

javax.net.ssl|ALL|23|org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1|2022-06-29 12:14:54.831 CEST|X509Authentication.java:246|No X.509 cert selected for RSA
javax.net.ssl|WARNING|23|org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1|2022-06-29 12:14:54.831 CEST|CertificateRequest.java:809|Unavailable authentication scheme: rsa_pkcs1_sha256

.............

    javax.net.ssl|DEBUG|23|org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1|2022-06-28 16:34:01.618 CEST|CertificateMessage.java:299|No X.509 certificate for client authentication, use empty Certificate message instead
    javax.net.ssl|DEBUG|23|org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1|2022-06-28 16:34:01.619 CEST|CertificateMessage.java:330|Produced client Certificate handshake message (
    "Certificates": <empty list>
    )

I have to be missing something about here that is not ok in the certificate but I can't figure out what. If anyone has any clue, I would love to hear it

Answer 1

Try to remove the key-store type parameter and alter your parameters something similar to

kafka.security.protocol=SSL
kafka.ssl.truststore.location=dir/xxxxx.jks
kafka.ssl.truststore.password=Password
kafka.ssl.keystore.location=dir/xxxxx.jks
kafka.ssl.keystore.password=Password
kafka.ssl.key.password=Password
kafka.ssl.enabled=true

Also, make sure your KAFKA Server is running on SSL.

Once you start the broker you should be able to see something like this in the server.log with addresses: PLAINTEXT -> EndPoint(192.168.64.1,9092,PLAINTEXT),SSL -> EndPoint(192.168.64.1,9093,SSL)

• To check quickly if the server keystore and truststore are setup properly you can run the following command

  openssl s_client -debug -connect localhost:9093 -tls1