How to compare data between two different DBMS?

Question

How to compare content of two selects from identical tables in two different DBMS? For example I want to compare results from Postgresql and Big Query. The result of queries is quite big (few million rows). I'm not sure whether exporting the data from one database and importing to temporary table in another is a good solution. What are common solutions to this kind of problem?

Answer 1

What you are looking for is a Data Validator tool. Refer to this open source tool developed by Google Cloud PSO Team

The Data Validation Tool is an open sourced Python CLI tool based on the Ibis framework that compares heterogeneous data source tables with multi-leveled validation functions.