Tables require a primary key when create a table with kafka topic

Question

I have a mysql table as this: I use kafka connector to add this table to kafka topic:

ksql> CREATE SOURCE CONNECTOR SOURCE_MYSQL_01 WITH (
    'connector.class' = 'io.debezium.connector.mysql.MySqlConnector',
    'database.hostname' = 'mysql',
    'database.port' = '3306',
    'database.user' = 'debezium',
    'database.password' = 'dbz',
    'database.server.id' = '42',
    'database.server.name' = 'asgard',
    'table.whitelist' = 'demo.customers',
    'database.history.kafka.bootstrap.servers' = 'kafka:29092',
    'database.history.kafka.topic' = 'dbhistory.demo' ,
    'include.schema.changes' = 'false',
    'transforms'= 'unwrap,extractkey',
    'transforms.unwrap.type'= 'io.debezium.transforms.ExtractNewRecordState',
    'transforms.extractkey.type'= 'org.apache.kafka.connect.transforms.ExtractField$Key',
    'transforms.extractkey.field'= 'id',
    'key.converter'= 'org.apache.kafka.connect.storage.StringConverter',
    'value.converter'= 'io.confluent.connect.avro.AvroConverter',
    'value.converter.schema.registry.url'= 'http://schema-registry:8081'
    );

and then I want to use this to create a table based on it:

CREATE TABLE  CUSTOMERS WITH (KAFKA_TOPIC='asgard.demo.CUSTOMERS', VALUE_FORMAT='AVRO');

then I get this error:

Tables require a PRIMARY KEY. Please define the PRIMARY KEY.
Use a partial schema to define the primary key and still load the value columns from the Schema Registry, for example:
        CREATE TABLE CUSTOMERS (ID INT PRIMARY KEY) WITH (...);

when I change this as suggested:

CREATE TABLE CUSTOMERS (id int primary key) WITH (KAFKA_TOPIC='asgard.demo.CUSTOMERS', VALUE_FORMAT='AVRO');

I get this error:

Duplicate column names: `ID`

did some search but still stuck here. what's wrong with this and how can I resolve it? Thanks

Answer 1

The issue is that the name you're assigning your PK column is clashing with the name of a column in the Avro schema being loaded from the schema registry.

You can name your key column whatever you like, as the column name is not persisted anywhere, so just name it something that doesn't clash, eg customer_id .

CREATE TABLE CUSTOMERS (customer_id int primary key) WITH (KAFKA_TOPIC='asgard.demo.CUSTOMERS', VALUE_FORMAT='AVRO');

Also notices that you have:

'key.converter'= 'org.apache.kafka.connect.storage.StringConverter'

Which I believe will be converting the key to a STRING . Not sure this is intentional. ksqlDB will work with the INT PK too...