使用docker将kafka主题数据写入redis

Question

I am using this repository kafka connect to redis.

Explain: What I want to do is to write kafka topics data into redis using docker. They have created a readme file to instruct how to set the configuration of the kafka:

curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors

connector.json file contains:

{
  "config" : {
    "name" : "RedisSinkConnector1",
    "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
    "tasks.max" : "1",
    "topics" : "mostafa"
  }
}

Problem: I know how to create a new topic in kafka, but the problem is where I don't know how to change the docker-compose or test the connection. While I have created a new topic in kafka, nothing shown in redis database!

I would be thankful if anyone could help me.

Answer 1

For starters, there is no Kafka Connect container in the compose file there, so you'll need to add one, or start Kafka Connect outside of Docker on your host machine.

Then, it's not clear if you've gotten the Redis connector properly loaded, so open up http://localhost:8083/connector-plugins to see if it is (this will also verify you've started the Connect Server}

Once that's done, you can post your config (you will need to remove the -s that hides the curl output). Once posted, you will want to be checking the logs of the running Connect process, or you can also go to http://localhost:8083/connectors/RedisSinkConnector1/status

Given what you've shown, and you got this far, both of the above probably say something about a Connection Exception to localhost:6379, since that's the default connection. You'll have to provide "redis.hosts": "redis:6379" as a property.

Then, also mentioned in the documentation

This connector expects records from Kafka to have a key and value that are stored as bytes or a string

So, it wouldn't hurt to also add key & value converter to your properties as well to specify the data types. If you are directly using the Connect container from Confluent, it's probably set to use the Avro converter, not string or bytes one

Here's an example of a valid configuration that you can POST

{
  "name" : "RedisSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
    "tasks.max" : "1",
    "topics" : "mostafa",
    "redis.hosts": "redis:6379",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter"
  }
}

With those adjustments, I would think sending any simple key-value message would work and then use redis-cli to run some scan/get key query

Answer 2

The following configuration would resolve the problem.

{
  "name" : "RedisSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
    "tasks.max" : "1",
    "topics" : "mostafa",
    "redis.hosts": "redis:6379",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter"
  }
}

add kafka-connect in docker-compose file

kafka-connect:
    hostname: kafka-connect
    image: confluentinc/cp-kafka-connect:latest
    container_name: kafka-connect
    ports:
      - 8083:8083
    depends_on:
      - schema-registry
      **- redis**
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:9092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: "quickstart-avro"
      CONNECT_CONFIG_STORAGE_TOPIC: "quickstart-avro-config"
      CONNECT_OFFSET_STORAGE_TOPIC: "quickstart-avro-offsets"
      CONNECT_STATUS_STORAGE_TOPIC: "quickstart-avro-status"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      **CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"**
      CONNECT_LOG4J_ROOT_LOGLEVEL: DEBUG
      CONNECT_PLUGIN_PATH: "/usr/share/java,/etc/kafka-connect/jars"
    volumes:
      - $PWD/jars:/etc/kafka-connect/jars

depends_on redis and CONNECT_REST_ADVERTISED_HOST_NAME variable very important in resolving this issue

Answer 3

Can you please send the soure connector details for RedisSinkConnector1

{ "name" : "RedisSinkConnector1", "config" : { "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector", "tasks.max" : "1", "topics" : "mostafa", "redis.hosts": "redis:6379", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter" } }