许可证主题的 Kafka Connect 复制因子

Question

I'm trying to run Kafka-Connect locally with docker-compose . Much as I like Confluent products and Kafka, it's sometimes a huge quest to pass some config or find one consistent example. In my docker-compose file, I'm using 6.0.0 versions for broker , zk , sr and kafka-connect right now, but I've tried older versions as well.

The broker( confluentinc/cp-server:6.0.0 ) fails with:

INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='_confluent-license', numPartitions=1, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='min.insync.replicas', value='2')], linkName=null, mirrorTopic=null) (kafka.server.AdminManager)

And I simply don't know how to pass confluent.topic.replication.factor as env vars to my workers. I've added both:

      CONNECT_CONFLUENT_TOPIC_REPLICATION_FACTOR: "1"
      CONFLUENT_TOPIC_REPLICATION_FACTOR: "1"

...but they are both ignored. What's more, I can't even find _confluent-license mentioned in docs anywhere, only _confluent-command .

How can I possibly make connect work locally inside docker-compose and without setting up 3 brokers?

Answer 1

The broker is failing, because that topic is created from it, so you want the variable on the broker

KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR: 1

Otherwise, you want cp-kafka , not cp-server

Answer 2

As OneCricketeer pointed out, the value must be set in the broker container, and that also explains why it's broker container that fails, not connect workers. That definitely makes sense. Broker can't fail because of a misconfigured clients.

What probably made it a lot more confusing to find a solution is that the env var and configuration property for license topic replication factor do not follow a regular pattern where env var is an uppercases configuration property with dots replaced with underscores(and prefix where applicable). In reality we get: confluent.topic.replication.factor -> KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR