如何使 Kafka Connect BigQuery Sink Connector 为每种事件类型而不是每个主题创建一个表？

Question

I'm using confluentinc/kafka-connect-bigquery on our Kafka (Avro) events. On some topics, we have more than one event type, eg, UserRegistered and UserDeleted are on the topic domain.user .

The subjects in our Schema Registry look as follows.

curl --silent -X GET http://avro-schema-registry.core-kafka.svc.cluster.local:8081/subjects | jq .

[...]
  "domain.user-com.acme.message_schema.domain.user.UserDeleted",
  "domain.user-com.acme.message_schema.domain.user.UserRegistered",
  "domain.user-com.acme.message_schema.type.domain.key.DefaultKey",
[...]

My properties/connector.properties (I'm using the quickstart folder.) looks as follows:

[...]
topics.regex=domain.*
sanitizeTopics=true
autoCreateTables=true
[...]

In BigQuery a table called domain_user is created. However, I would like to have two tables, eg, domain_user_userregistered and domain_user_userdeleted or similar, because the schemas of these two event types are quite different. How can I achieve this?

Answer 1

I think you can use the SchemaNameToTopic Single Message Transform to do this. By setting the topic name as the schema name this will propagate through to the name given to the created BigQuery table.