Kafka connect JDBC sink connector - tables without Primary Key

Question

I want to replicate 100+ tables from old MySQL server to PostgreSQL. I have setup debezium which is running fine. Now I want to setup a JDBC sink connector to PostgreSQL. Also I need to enable the DELETEs.

In this case, how can I configure my sink connector for the tables without a primary key?

It should replicate Insert, Update, and delete.

Answer 1

I have used Debezium and Kafka Connect 2 Postgres. I have specified the key column pk.mode=record_key and pk.fields=name_of_pk_column , because it's needed by Kafka Connect, so it can delete ( enable.delete=true ) and update. Also you can set auto.evolve=true.

If you set enable.delete to true, you have to specify the PK. The column key must be the same in source and in destination table.

But this is not handy if you need to transfer more than few tables.

I did not tried it, but I think a single message transformation script might obratin the key from kafka message, transform it (maybe rename it) and us it in kafka_connect_postgres settings.

I actually have 1:1 databases on boths sides, so getting pk column name I have to only worry about.

If you desire, I can provide mine settings for Kafka connect, Kafka pg connect...

connector-pg.properties
name=sink-postgres
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
#tasks.max=2
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
topics=pokladna.public.books #my debezium topic with CDC from source DB
connection.url=jdbc:postgresql://localhost:5434/postgres
connection.user=postgres
connection.password=postgres
dialect.name=PostgreSqlDatabaseDialect
table.name.format=books_kafka #targe table in same db, in thus case given by url
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.drop.tombstones=false
auto.create=true
auto.evolve=true
insert.mode=upsert #insert does not work for updates
delete.enabled=true
pk.fields=id #pk column name same in target and resource db
pk.mode=record_key

try to use SMT extract field , but for me it didn't worked yet. I still have no clue how to extract the primary key from kafka message automatically. Maybe I have to write my own SMT...

Actually I'm gonna start a new SO topic about it. Haha