Best practice for auditing data in SQL Server and retrieving point in time data

Question

I've been doing history tables for some time now in databases, but never put too much effort or thought into it. I wonder what is the best practice out there.

My main goal is to record any changes to a record for a particular day. If more than one change happens in a day then then only one history record will exist. I need to record the date the record was changed, also when I retrieve data I need to pull the correct record from history as it was at a particular time. So for example I have a customers table and want to pull out what their address was for a particular date. My Sprocs like get Cust details will take in an optional date and if no date is passed in then it returns the most recent record.

So here's what I was looking for advice on:

Do I keep the history table in the same table and use a logical delete flag to hide the historical ones? I normally don't do this as some tables can change a lot and have lots of records. Do I use a separate table that mirrors the main table? I usually do this. Should I only put change records into the history table and not the current one? What is the most efficient way given a date to pull out the right record at a point in time, get every record for a customer <= date passed in, and then sort by most recent date and take the top?

Thanks for all the help... regards M

Answer 1

Suggestion is to use trigger based auditing and create triggers for all tables you need to audit.

With triggers you can accomplish the requirement for not storing more than one record update per day.

I'd suggest you check out ApexSQL Audit that generates triggers for you and try to reverse engineer what triggers they use, how storage tables look like and such.

This will give you a good start and you can work form there.

Disclaimer: not affiliated with ApexSQL but I do use their tools on a daily basis.

Answer 2

I'm no expert in the field but a good sql consultant once told me that a good aproach is generally to use the same table if all data can be changed. Otherwise have the original table contain only core nonchangable data and the historical table contain only stuff that can be changed.

Answer 3

You should defintely read this article on managing bitemporal data . The nice thing about this approach is it enables an auditable way of correcting historical data.

I beleive this will address your concerns about modidying the history data

Answer 4

I've always used a modified version of the audit table described in this article . While it does require you to pivot data so that it resembles your table's native structure, it is resilient against changes to the schema.

You can create a UDF that returns a table and accepts a table name ( varchar ) and point in time ( datetime ) as parameters. The UDF should rebuild the table using the audit (historical values) and give you the effective values at that date & time.