1 个主题映射到两个不同的数据库表 Kafka Sink Connector

Question

I am currently having trouble mapping my Kafka topic: st_record to two separate database table: 1) gt_school.strecord_1week 2) gt_school.strecord_1semester. My Kafka sink configuration is

"tasks.max": "1",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": " "'"$URL"'",
"topics":"st_record",
"table.name.format": "gt_school.strecord_1week, gt_school.strecord_1semester",
"table.whitelist": "gt_school.strecord_1week, gt_school.strecord_1semester",
"transforms":"route",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter,
"transforms.route.regex":"st_record",
"transforms.route.replacement":"gt_school.strecord_1week, gt_school.strecord_1semester"

I tried table.name.format, table.whitelist, and transform route however everytime I received the following error that both tables are unfound

io.confluent.connect.jdbc.sink.TableAlterOrCreateException: Table "gt_school"."strecord_1week, gt_school"."strecord_1semester" is missing and auto-creation is disabled"

Which is true, it should return in this format, "gt_school.strecord_1week, gt_school.strecord_1semester". Does anyone know what field it should map the two tables to from 1 topic name. Am I suppose to use table.name.format. I know that in default the topic and table name are suppose to be the same however I route it and still errors

Answer 1

The error only says one table isn't found. Not two. The comma is within quotes... JDBC sink only writes to one table, per topic. Plus, tables cannot contain commas, as far as I know.

RegexRouter doesn't split your topic into two. It only renames the topic to a static string.

If you want to write to two distinct tables, create two separate connectors with

"topics":"st_record",
...
"transforms":"route",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":".*",
"transforms.route.replacement":"$0_1week"

"topics":"st_record",
...
"transforms":"route",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":".*",
"transforms.route.replacement":"$0_1semester"

However, this will obviously duplicate data in the database, so I'd recommend creating one table with data from the topic, then two VIEWs instead to do different queries of weeks/semesters