Unable to run kafka connect datagen inside kafka connect docker image

Question

I am trying to run kafka datagen connector inside kafka-connect container and my kafka resides in AWS MSK using: https://github.com/confluentinc/kafka-connect-datagen/blob/master/Dockerfile-confluenthub .

I am using kafdrop as a web browser for kafka broker (MSK). I don't see Kafka datagen generating any test messages. Is there anything other configuration I need to do except installing the kafka-datagen connector

Also, how can I check inside confluentinc/kafka-connect image what topics are created and whether messages are consumed or not?

Dockerfile looks like:

ARG BASE_PREFIX=confluentinc 
ARG CONNECT_IMAGE=cp-kafka-connect

FROM $BASE_PREFIX/$CONNECT_IMAGE:6.1.0

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.4.0

docker exec 51e32e20b292 bash -c 'echo $CONNECT_PLUGIN_PATH'                                           
shows : /usr/share/java,/usr/share/confluent-hub-components

[appuser@88db8385b575 ~]$ ls -la /usr/share/confluent-hub-components/
total 20
drwxr-xr-x 1 appuser appuser 4096 Mar 26 21:19 .
drwxr-xr-x 1 root    root    4096 Feb  4 21:10 ..
drwxr-xr-x 6 appuser appuser 4096 Mar 26 18:00 confluentinc-kafka-connect-datagen

[appuser@88db8385b575 ~]$ ls -la /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen/
total 28
drwxr-xr-x 6 appuser appuser 4096 Mar 26 18:00 .
drwxr-xr-x 1 appuser appuser 4096 Mar 26 21:19 ..
drwxr-xr-x 2 appuser appuser 4096 Mar 26 18:00 assets
drwxr-xr-x 4 appuser appuser 4096 Mar 26 18:00 doc
drwxr-xr-x 2 appuser appuser 4096 Mar 26 18:00 etc
drwxr-xr-x 2 appuser appuser 4096 Mar 26 18:00 lib
-rw-r--r-- 1 appuser appuser 1380 Mar 26 18:00 manifest.json

Docker logs:

docker logs 51e32e20b292 | grep "DatagenConnector"  
        
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",

Answer 1

I just added in the dockerfile and ran RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.4.0 inside the dockerfile. Nothing else. No error logs.

That alone doesn't run the connector, only makes it available to the Connect API. Notice the curl example in the docs https://github.com/confluentinc/kafka-connect-datagen#run-connector-in-docker-compose

So, expose port 8083 and make the request to add the connector, and make sure to add all the relevant environment variables when you're running the container