Facing issue while configuring Confluent kafka with Azure databricks

Question

I am newbie to Azure databrciks and this forum. I am actually carrying out exercise for steaming Confluent Kafka on Azure databricks. I am able to stream the content on spark. However, there is slight problem as I am exposing username and password in the program. I would rather passed this through the variable(or Azure key-vault). I have tried passing username and password using variable but this approach is not working. I am getting error saying that 'unable to create Kafka consumer'. Can you please let me know how I can proceed? I am using scala for this.

val streamingInputDF = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", host)
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"AAAAAA\" password=\BBBBB\";")
.option("kafka.ssl.endpoint.identification.algorithm", "https")
.option("kafka.sasl.mechanism", "PLAIN")
.option("startingOffsets", "earliest")
.option("failOnDataLoss", "false")
.option("subscribe", "Test")
.load()

This is how I want to pass but not working:-

val streamingInputDF = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", host)
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.jaas.config", f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=$username password=$password";")
.option("kafka.ssl.endpoint.identification.algorithm", "https")
.option("kafka.sasl.mechanism", "PLAIN")
.option("startingOffsets", "earliest")
.option("failOnDataLoss", "false")
.option("subscribe", "streaming_test_6")
.load()

Answer 1

You need to check below points:

1. Make sure you have kafka-clients jar in your class path.

2. Make sure you have compatible version of kafka and spark installed

You can refer article by ANGELA CHU, GIANLUCA NATALI AND CAIO MORENO, where detailed description is given.

Scala Notebook link

Answer 2

Thank you all who tried to provide suggestion. I am able to resolve issue. Below is the line where changes are required

.option("kafka.sasl.jaas.config",
   "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"" + confluentApiKey + "\"password=\"" + confluentSecret + "\";")