How to use postgres JDBC sink connector to stream to mysql connector

Question

Happy Thursday! I've been experimenting with creating a connector Postgres Connector in Debezium but I can only capture the changes if the table already exists in my MySQL instance which isn't ideal. Because then I would have to write a script in Python that would handle such events and it may be easier to use something that already exists than reinvent the wheel. I want to be able to capture the DDL in the actual connector. I came across this blog post. https://debezium.io/blog/2017/09/25/streaming-to-another-database/ and I got it working on my local set-up which is great, but the only issue is I want to go in the opposite direction. (I am able to capture new records, deleted records, and updated records, and it creates the new tables and new columns as well if they don't exist). I want to stream from postgres and have the connector insert into a target db in mysql. I tried switching the jdbc source and sink connectors respectively but I wasn't getting the new records inserted from postgres into mysql. It seems like I can find people inserting into postgres from mysql all over the place but not the other direction. Here is the GitHub directory I based my set up off of to get the mysql-kafka-postgres to work. https://github.com/debezium/debezium-examples/tree/main/unwrap-smt

Answer 1

I believe there are two different questions here:

1. How to handle non existing columns in Mysql. The JDBC sink connector should have a flag called auto.create that, if set to true allows the connector to create tables if they don't exist ( auto.evolve also allows table evolution)

2. PG -> Kafka -> Mysql is possible, you can find an example of it that I wrote some time ago here . The examples uses Aiven for PostgreSQL and Aiven for Apache Kafka but you should be able to adapt the connectors to work in any kind of PG and Kafka.

Would be interesting to know there your PG->Kafka->MySQL pipeline stops working.

Disclaimer: I work for Aiven