在SQL中保存大型每月数据备份的最佳方法是什么？

Question

I work on a program that stores information about network connections across my University and I have been asked to create a report that shows the status changes of these connections over time. I was thinking about adding another table that has the current connection information and the date the data was added so when the report is run, it just grabs the data at that date, but I'm worried that the report might get slow after a couple of months as it would be adding about 50,000 rows every month. Is there a better way to do this? We use a Microsoft SQL Server.

Answer 1

It depends on the reason you are holding historical data for facts.

If the reason is:

• For reporting needs then you could hold it in the same table by adding two date columns FromDate and ToDate which will remove the need to join the active and historical data tables later on.
• Just for reference then it makes sense to have it in a different table as it may decrease the performance of your indexes on your active table.

I'll highlight the Slowly Changing Dimension (SCD) type 2 approach that tracks data history by maintaining multiple versions of records and uses either the EndDate or a flag to identify the active record. This method allows tracking any number of historical records as each time a new record is inserted, the older ones are populated with an EndDate .

Step 1: For re-loaded facts UPDATE IsActive = 0 for the record to be history preserved and populate EndDate as the current date.

merge ActiveTable as T
using DataToBeLoaded as D  
    on  T.ID = D.ID
         and 
        T.isactive = 1  -- Current active entry
  when matched then
    update set T.IsActive = 0,
               T.EndDate = GETDATE();

Step 2: Insert the latest data into the ActiveTable with IsActive = 1 and FromDate as the current date.

Disclaimer: The following approach using SCD 2 could make your data warehouse huge. However, I don't believe it would affect performance much for your scenario.