Kafka/questDB JDBC Sink Connector: tables not created using "topics.regex"

Question

I am using "confluentinc/kafka-connect-jdbc:10.2.6" as my JDBC connector to transfer Kafka topics into my questDB.

When I provide explicit topic names it is working as expected. But when I use topic names based on regex then it's not working, the tables are not being created in my questDB database.

What am I missing in my JDBC settings?

Thanks!

explicit version (working):

{
  "name": "jdbc_sink_ftx",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "trades-FTX-ETH-USD-PERP, trades-FTX-FTM-USD-PERP",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}

regex-version (not working):

{
  "name": "jdbc_sink_ftx_regex",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics.regex": "FTX",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}

Answer 1

It would be best to check QuestDB logs. Grepping for ' E ' should show errors that usually include description and OS error code in this case. Topic names in Kafka Connector are used as table names. QuestDB uses table names as directories on disk and validates them for security purposes. '.', '/' and '' are not allowed characters in table name. Specific OS can also apply additional restrictions on file names.

It would be best to check QuestDB logs though.

Answer 2

The regex needs to match with the complete topic names like so:

"topics.regex": "trades-FTX-.*"