[英]How to edit replication factor when Debezium creates a new topic
I am using kafka-connect + Debezium to capture data changes in my MySQL cluster.我正在使用 kafka-connect + Debezium 来捕获我的 MySQL 集群中的数据更改。 Currently we use Debezium's table.whitelist
to ingest only a very small subset of tables in that MySQL cluster.目前,我们使用 Debezium 的table.whitelist
仅摄取 MySQL 集群中的一小部分表。 Whenever I add a new table to the table.whitelist
config, Debezium will create a new kafka topic that only uses replication_factor = 2
and partition=1
.每当我向table.whitelist
配置添加新表时,Debezium 将创建一个仅使用replication_factor = 2
和partition=1
的新 kafka 主题。 My Kafka cluster has a total of 4 brokers (1 leader and 3 followers) and in my /opt/kafka-connect/worker.properties
file I have set the following:我的 Kafka 集群共有 4 个代理(1 个领导者和 3 个追随者),在我的/opt/kafka-connect/worker.properties
文件中,我设置了以下内容:
offset.storage.replication.factor=4
config.storage.replication.factor=4
status.storage.replication.factor=4
This results in me having to manually re-balance the topic every time I add a new table and it gets old real fast.这导致我每次添加新表时都必须手动重新平衡主题,并且它会很快变旧。 The last entry is the new MySQL table that was recently added, notice that it is replicated on 2 brokers while the other topics are replicated on 3.最后一个条目是最近添加的新 MySQL 表,请注意它在 2 个代理上复制,而其他主题在 3 个上复制。
Am I editing the wrong config file or using the wrong settings?我是在编辑错误的配置文件还是使用了错误的设置? Please halp!请停下来!
Below is my Debezium config下面是我的 Debezium 配置
{
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"snapshot.locking.mode": "none",
"max.queue.size": "300000",
"tasks.max": "8",
"database.history.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"<USER>\" password=\"<PASSWORD>\";",
"database.history.kafka.topic": "dbhistory_mysql",
"database.history.consumer.ssl.truststore.password": "<PASSWORD>",
"database.history.consumer.security.protocol": "SASL_SSL",
"database.history.consumer.ssl.truststore.location": "/PATH/TO/TRUSTSTORE.jks",
"table.whitelist": "schema.table1,schema.table2,schema.table3,schema.table4,schema.table5",
"decimal.handling.mode": "string",
"database.history.kafka.recovery.poll.interval.ms": "30000",
"database.history.producer.sasl.mechanism": "PLAIN",
"database.history.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"<USER>\" password=\"<PASSWORD>\";",
"database.user": "debezium",
"database.server.id": "123",
"database.history.producer.security.protocol": "SASL_SSL",
"database.history.kafka.bootstrap.servers": "broker1.com:9092,broker2.com:9092,broker3.com:9092,broker4.com:9092",
"database.history.producer.ssl.truststore.location": "/PATH/TO/TRUSTSTORE.jks",
"database.server.name": "mysql",
"database.port": "3306",
"database.serverTimezone": "US/Pacific",
"database.history.producer.ssl.truststore.password": "<PASSWORD>",
"database.hostname": "mysql.cluster.amazonaws.com",
"database.password": "<PASSWORD>",
"name": "mysql-connector",
"max.batch.size": "20480",
"database.history.consumer.sasl.mechanism": "PLAIN"
}
Those three properties are internal Connect topics, not auto created user topics.这三个属性是内部 Connect 主题,而不是自动创建的用户主题。
For auto-created topics (which is recommended to have disabled), you set default partition and replication factors in the broker server.properties对于自动创建的主题(建议禁用),您在代理 server.properties 中设置默认分区和复制因子
num.partitions
default.replication.factor
I believe these are also dynamic variables that can be overriden using kafka-configs
tooling... Maybe with --alter --entity-type brokers --entity-default
... But probably easier to reason about if the property is actually in a config file我相信这些也是可以使用kafka-configs
工具覆盖的动态变量...也许使用--alter --entity-type brokers --entity-default
...但可能更容易推断出该属性是否实际上在一个配置文件
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.