[英]Configure a debezium connector for multiple tables in a database
I'm trying to configure a Debezium connector for multiple tables in a MySQL database (i'm using debezium 1.4 on a MySQL 8.0).我正在尝试为 MySQL 数据库中的多个表配置 Debezium 连接器(我在 MySQL 8.0 上使用 debezium 1.4)。 My company have a nomenclature pattern to follow when creating topics in kafka, and this pattern does not allow the use of underscores (_), so I had to replace them with hyphens (-)
在kafka中创建主题时,我的公司有一个命名模式要遵循,并且这种模式不允许使用下划线(_),所以我不得不用连字符(-)替换它们
So, my topics names are:所以,我的主题名称是:
Topic 1话题一
fjf.db.top-domain.domain.sub-domain.transaction-search.order-status
WHERE
- transaction-search = schema "transaction_search"
- order-status = table "order_status".
- All changes in that table, must go to that topic.
Topic 2话题二
fjf.db.top-domain.domain.sub-domain.transaction-search.shipping-tracking
WHERE
- transaction-search = schema "transaction_search"
- shipping-tracking = table "shipping_tracking"
- All changes in that table, must go to that topic.
Topic 3话题 3
fjf.db.top-domain.domain.sub-domain.transaction-search.proposal
WHERE
- transaction-search = schema "transaction_search"
- proposal = table "proposal"
- All changes in that table, must go to that topic.
I'm trying to use the transforms "ByLogicalTableRouter", but i can't find a regex solution that solve my case.我正在尝试使用转换“ByLogicalTableRouter”,但我找不到解决我的情况的正则表达式解决方案。
{ "name": "debezium.connector",
"config":
{
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "myhostname",
"database.port": "3306",
"database.user": "debezium",
"database.password": "password",
"database.server.id": "1000",
"database.server.name": "fjf.db.top-domain.domain.sub-domain.transaction-search",
"schema.include.list": "transaction_search",
"table.include.list": "transaction_search.order_status,transaction_search.shipping_tracking,transaction_search.proposal",
"database.history.kafka.bootstrap.servers": "kafka.intranet:9097",
"database.history.kafka.topic": "fjf.db.top-domain.domain.sub-domain.transaction-search.schema-history",
"snapshot.mode": "schema_only",
"transforms":"RerouteName,RerouteUnderscore",
"transforms.RerouteName.type":"io.debezium.transforms.ByLogicalTableRouter",
"transforms.RerouteName.topic.regex":"(.*)transaction_search(.*)",
"transforms.RerouteName.topic.replacement": "$1$2"
"transforms.RerouteUnderscore.type":"io.debezium.transforms.ByLogicalTableRouter",
"transforms.RerouteUnderscore.topic.regex":"(.*)_(.*)",
"transforms.RerouteUnderscore.topic.replacement": "$1-$2"
}
}
But with that, I'm getting the error below, which indicates that it is trying to send everything to the same topic但是有了这个,我得到了下面的错误,这表明它正在尝试将所有内容发送到同一个主题
Caused by: org.apache.kafka.connect.errors.SchemaBuilderException: Cannot create field because of field name duplication __dbz__physicalTableIdentifier
How can i make a transform that will forward the events of each table to their respective topic?如何进行转换,将每个表的事件转发到各自的主题?
In the first transforms,im trying to remove the duplicated schema name in the topic routering.
在第一次转换中,我试图删除主题路由中重复的模式名称。
After transforamtion with your regex you'll have two dots, so you need to fix it:使用正则表达式进行转换后,您将有两个点,因此您需要修复它:
"transforms.RerouteName.topic.regex":"([^.]+)\\.transaction_search\\.([^.]+)",
"transforms.RerouteName.topic.replacement": "$1.$2"
You can try to use ChangeCase SMT from Kafka Connect Common Transformations .您可以尝试使用来自Kafka Connect Common Transformations的ChangeCase SMT。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.