Is there a data integrity test software?

Question

We have large amounts of data in multiple mysql databases which is constantly updated from external sources. Is there some software (preferably php based) with which we can define rules to test against the database, for example regular expressions on the data, frequency of updates, missing data etc..) and run checks regularly reporting that something has gone wrong or a trend has changed in the data ?

Answer 1

How about STFW? Googling for "Mysql data quality" brought (among others) a link to http://www.talend.com

Otherwise, I'd have a look at data warehousing tools - Oracle Warehouse Builder for example has some mechanisms for data auditing.

Kind regards, Frank

Answer 2

If you have multiple db tables that are not joined with foreign keys, then you should add and use them for data integrity.

If you have lots of PL/SQL code then you need unit tests for it (yes, DB needs tests too). So in the end you'll end up with "continuous integration" that runs your tests periodically. And yes, you have to write it yourself

See http://www.slideshare.net/antonkeks/database-refactoring for more info.

If you have to sync databases, then i'd recommend SQLYog.

Answer 3

If you have properly designed the database you don't have many data integrity problems. This means doing the work of setting up PK/FK relationships, data constraints, correct datatypes, triggers, etc. This especially means you never consider that the application will handle all that. It might mean setting up jobs to check on certain types of data entry and notifying someone of possible problems. It might mean revising all your data imports to use a standard set of cleaning routines. It might mean creating a way to identfy and merge duplicate records (all complex databases should havea deduping application written so that the users can make chocies about what data to keep and what data to save when duplicates are found).

If you didn't design the database correctly, you need to set these things up in the database one at a time depending on your business rules, fixing the bad data as you go. There is no easy solution for the failure of the developers to design properly.

Since the needs of each database are very different, no one that I know of has automated a way to enforce all integrity rules, this is a large part of what the database designer does when designing the database. I ceratinly wouldn't trust any COTS program to do it either based on how badly designed every COTS database I have ever had the displeaure to support has been.