How to create the JDBC sink connector with multiple topic using topic regex option

Question

Created a JDBC source connector

catalog.pattern = test_01

source connector configuration

    {
  "name": "jdbcsource",
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "catalog.pattern": "test_01",
  "mode": "timestamp",
  "timestamp.column.name": "UpdateDate",
  "topic.prefix": "jdbcsource-"
}

Under test_01 database example01 (parent table) examplerole (child table) with foreign reference as user_id(refer the schema below)

  mysql> describe example01;
+------------------+--------------+------+-----+-------------------+-----------------------------+
| Field            | Type         | Null | Key | Default           | Extra                       |
+------------------+--------------+------+-----+-------------------+-----------------------------+
| user_id          | int(11)      | NO   | PRI | NULL              | auto_increment              |
| user_name        | varchar(255) | NO   |     | NULL              |                             |
| user_description | text         | YES  |     | NULL              |                             |
| CreationDate     | timestamp    | NO   |     | CURRENT_TIMESTAMP |                             |
| UpdateDate       | timestamp    | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+------------------+--------------+------+-----+-------------------+-----------------------------+
5 rows in set (0.00 sec)

mysql> describe examplerole;
+--------------+---------------+------+-----+-------------------+-----------------------------+
| Field        | Type          | Null | Key | Default           | Extra                       |
+--------------+---------------+------+-----+-------------------+-----------------------------+
| prd_id       | int(11)       | NO   | PRI | NULL              | auto_increment              |
| prd_name     | varchar(355)  | NO   |     | NULL              |                             |
| prd_price    | decimal(10,0) | YES  |     | NULL              |                             |
| user_id      | int(11)       | NO   | MUL | NULL              |                             |
| CreationDate | timestamp     | NO   |     | CURRENT_TIMESTAMP |                             |
| UpdateDate   | timestamp     | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+--------------+---------------+------+-----+-------------------+-----------------------------+
6 rows in set (0.00 sec)

While creating sink connector with topics.regex option with key reference in the table

curl -XPOST -H 'Accept: application/json' -H "Content-type: application/json" -d '{
    "name": "MySQL-JDBC-sink-connector",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "tasks.max": "1",
    "topics.regex": "jdbcsource*",
    "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
     "connection.user": "root",
    "connection.password": "********",
    "insert.mode": "insert",
    "pk.mode": "record_value",
    "pk.fields": "user_id",
    "auto.create": "true",
     "auto.evolve": "true"

}
}' 'localhost:8083/connectors'
echo "\n"

error for above sink configuration, but topics regex option is given in the configs

[2019-06-10 04:54:56,678] ERROR Uncaught exception in REST call to /connector-plugins/io.confluent.connect.jdbc.JdbcSinkConnector/config/validate (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)
org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex
    at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:96)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:269)
    at org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource.validateConfigs(ConnectorPluginsResource.java:81)
    at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.

Tried creating multiple jdbc sink connector with option topics:<t1,t2> refer below configuration

{
  "name": "jdbcsink",
  "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
  "topics": [
    "jdbcsource-example01",
    "jdbcsource-examplerole"
  ],
  "connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
  "connection.user": "root",
  "connection.password": "********",
  "auto.create": "true",
  "auto.evolve": "true"
}

Getting below error

[2019-06-10 04:53:19,738] WARN [Consumer clientId=consumer-6, groupId=connect-jdbcsink] Error while fetching metadata with correlation id 2 : { jdbcsource-examplerole=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.common.errors.InvalidTopicException: Invalid topics: [ jdbcsource-examplerole]
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2019-06-10 04:53:19,740] INFO Stopping task (io.confluent.connect.jdbc.sink.JdbcSinkTask)
[2019-06-10 04:53:19,749] INFO After filtering the tables are: `test_01`.`example01`,`test_01`.`examplerole` (io.confluent.connect.jdbc.source.TableMonitorThread)

Tried creating the sink connector with an individual topic, I can able to create the sink connector. Facing the above issues while creating multiple sink connectors in a single config

Kindly suggest the configuration option for JDBC multiple sink connector creations using topics.regex and auto.create option

Answer 1

If you're specifying topics.regex , then don't specify topics . Instead of

   "topics": [],
   "topics.regex": "example*",

you just need

   "topics.regex": "example*",