来自多个MySQL数据库的ETL

Question

I have a system which extracts data from multiple MySQL databases with varying schemas, and performs many queries (with joins) and loads the output to another MySQL database.

These queries were once just a quick fix, but they've grown to over 10000 lines and thus choke the source databases.

I'm designing an efficient ETL pipeline by analyzing the SQL queries, but is there any temporary fix such as a tool which could analyze the queries and reduce the number of steps to reach the required schema?

Any help would be life-saving :)

Answer 1

Rather than perform a query on many MySQL databases (which are optimized for writes), you should move all of your queries to the Redshift database (which is optimized for reads).

But to do this, you need the data. Look into an ETL service that will clone ALL the data over to your Redshift. We use Stitch Data but there are many players in the space. You can set up multiple integrations, such that each MySQL db is pumping data into the same Redshift db (I'd recommend setting each up under a uniquely-named schema).

Once the data are all loaded, you can run your various queries in AWS Data Pipelines to create derived tables. Each query can be each own job, that way you can monitor and modify on a per-query basis.