Confluent-hub failed to download plugins 'Component not found'

Question

I am using kafka/kafka-connect via confluent platform helm-chart ( here ).

For Kafka-Connect I want to install both debezium SQLServer source connector and BQ sink connector, the only way I found it with the current setup is to update the template/deployment.yaml to add an initContainers section to download the plugins using confluent-hub CLI.

The problem is, the confluent-hub command is inconsistent, sometimes download the plugins and sometimes not, and when asked on confluent community on slack, I found others face the same problem.

Answer 1

The solution I implemented for such case is to retry during the initContainer script till the plugin is downloaded using the following script -it assumes that confluent-hub is just the bin directory away, and the connector config file under the path of --worker-configs -.

For those who don't know what does the $? do, it holds the response code of the last ran command, which is in our case is the ./bin/confluent-hub install... command. ( more details )

./bin/confluent-hub install wepay/kafka-connect-bigquery:2.4.2 --component-dir bqkc  --worker-configs /etc/kafka/connect-standalone.properties --no-prompt
while [ $? -eq 1 ]
do
    echo "Failed to download the connector, will sleep and retry again"
    sleep 10
    ./bin/confluent-hub install wepay/kafka-connect-bigquery:2.4.2 --component-dir bqkc  --worker-configs /etc/kafka/connect-standalone.properties --no-prompt
done