MySQL 数据库到 BigQuery 的传输

Question

I want to create connector (something like Debezium in Kafka-Connect) to reflect every change in MySQL source database in BigQuery tables.

There is one problem - source database is dropped every 10mins and re-created - some of rows are the same, some are updated and some are totally new. So I cannot do it via Debezium, because every 10min I would have all records in Kafka.

I want migrate to BQ tables just new or updated values. The mechanism to "copy" the whole source database but to deduplicate records (which not be excatly the same because this will be a new database). So for example create hash from every record and check - if hash is already in BQ - pass and if there is not add it.

I think it should be this:

Best effort de-duplication

but how to create the whole pipeline with MySQL as a source.

Answer 1

Cloud Data Fusion Replication lets you replicate your data continuously and in real time from operational data stores, such as SQL Server and MySQL, into BigQuery.
To use Replication, you create a new instance of Cloud Data Fusion and add the Replication app.

In short you do below

1. Set up your MySQL database to enable replication.
2. Create and run a Cloud Data Fusion Replication pipeline.
3. View the results in BigQuery.

Yu can see more at Replicating data from MySQL to BigQuery