KAFKA SSL connectivity using pem key and client certificate

Question

I am able to connect to kafka and read data from CLI (bin/kafka-console-consumer.sh) using below ssl details in client.properties

ssl.keystore.location=/test/keystore.jks
ssl.keystore.password=abcd1234
ssl.key.password=abcd1234

Command: bin/kafka-console-consumer.sh --bootstrap-server 'server details'  --topic topic_name --consumer.config client.properties --group group-id

But I am unable to connect from python or spark using the same data

consumer = KafkaConsumer(topic,bootstrap_servers=bootstrap_server,security_protocol='SSL',sasl_mechanism='PLAIN',ssl_certfile='certificate.pem',ssl_keyfile='pk.key')

I tried changing multiple options in the above code, like adding check_host_name etc, but no luck. The kafka is not owned by our teams, it a different team who manages it and when we request access we get a private key and certificate along with CA bundle and ARN name.

From Spark(Python), I tried below code

sdf1 = spark.readStream.format("kafka")
       .option("kafka.bootstrap.servers",bootstrap_server)
       .option("subscribe", topic_name)
       .option("startingOffsets", "latest")
       .option("kafka.security.protocol","SSL")
       .option("kafka.ssl.keystore.location",'keystore.jks')
       .option("kafka.ssl.keystore.password", '****')
       .option("kafka.ssl.key.password",'****')
       .load()

I am getting error like "org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-kafka-source-xxxxxxx-xxxxx-xxxxx"

The above error would be related to spark generating unique group id everytime it is accesseing. Usage of group-id in spark dataframe is allowed only in spark 3.0 and above. I need option to fix this in spark 2.4.4.

Any suggestions would be appreciated.

Answer 1

You just need to give the principal you are using to authenticate access to the topic regardless of the consumer group. It would look like this:

kafka-acls --authorizer-properties zookeeper.connect=zk_ip_or_fqdn:2181  --add  --allow-principal User:"userName" --operation All --topic yourTopicName --group=*

userName (principal name) in your case will be the subject name of your SSL certificate, in the form of "CN=toto,OU=titi,...".