比较来自不同数据库的两行的列

Question

I'm trying to validate that a data transfer process is working correctly. I have source tables in one database and destination tables in a different database. I want to validate the data transfer for a specific key value.

Ultimately I want results back in the following type of format:

Table     Column     Matches
----------------------------
Company   Name       Y
Company   Address    Y
Company   Phone      N

I need to keep the list of columns dynamic so that this code doesn't change if a column is added to one of the tables. The list of tables is known.

Right now I'm using cursors. One to loop through a list of the tables that I need to compare and another for each table that loops through a query returning the list of columns. It works, but I'm concerned about the performance. I'm examining 22 tables and might need to compare multiple records in a table for the specified key value. So, comparing all of the records for one key right now instantiates 30-40 cursors.

I feel like there should be a better solution but haven't been able to find anything that does the job while remaining as dynamic as possible.

Does anyone have any ideas for me to try? Thanks in advance!

Answer 1

It turns out that the fastest way for me to do this was using C#. I just pulled all of the records to be analyzed into data tables in C# and analyzed them from memory. It reduced my total time to under 2s for each key value analyzed. And that includes parsing the output to XML in the way that I want. At this point I'm feeling like that's about as fast as I can expect it to be. Thanks to everyone for their suggestions.

Answer 2

SQL only: You need to extract the id and column from each database's table and create exclusion sets and merge them. Then you can use count() to figure out if there are any that are mismatched.

I haven't done SQL in ages but maybe this will be enough pseudocode to start you up:

SELECT count(1) from (
    (SELECT id, columnname FROM tablename EXCEPT SELECT id, columnname FROM othertable) 
        UNION
    (SELECT id, columnname FROM othertable EXCEPT SELECT id, columnname FROM tablename)
)

You could also do inner join and compare missing records twice which would be a bit faster but more complex.

Answer 3

Iterations in SQL Server are very poor and you should avoid them whenever possible. I'd rather go "relational". For example you can compare the table with a FULL OUTER JOIN and check if any of the keys are NULL . Here, obviously, if you add a column you have to change your query slightly.

If you want to use cursors I'd suggest you make them readonly and fast forward and see if you gain something. Like this:

DECLARE C CURSOR FAST_FORWARD FOR
...
...
...