数据仓库-数据版本控制

Question

I am currently designing a data warehouse for a financial company. While a large amount of the process is quite standard I have been presented with an issue (That I believe only exists in the finance sector) of data events that effect a number of rows and their history that can happen at any time.

To explain the issue better. Assume we have an Account A and other 2 months 4 transactions have occurred that effect its balance, changing it from 10000 to 20000. When I run a report for that month its fine it will show the activity that derives that value. Now it gets difficult, a month after I backdate a transaction that effects that balance changing it from 20000 to 15000.

Running a report Before that back dating should tell me the original 20000, but after back dated transaction should tell me the 15,000.

To illustrate better refer to data below.

---

Transactions for September and October

with a back dated transaction on the 28th of October for the 13th of September of $500

and a back dated transaction on the 8th of November for the 17th of September to credit the $-50

╔═════════════════╦═════════════════════════╦════════╦══════════════════╦═══════════════╦═════════════╦═════════╗
║ Key_Transaction ║ SK_TransactionEffective ║ Amount ║ PrincipleBalance ║ SK_ReportDate ║ SK_AsOfDate ║ Version ║
╠═════════════════╬═════════════════════════╬════════╬══════════════════╬═══════════════╬═════════════╬═════════╣
║               1 ║ 12/09/2018              ║  -1000 ║            20000 ║ 12/09/2018    ║ NULL        ║ 1       ║
║               6 ║ 13/09/2018              ║   -500 ║            19500 ║ 13/09/2018    ║ 28/10/2018  ║ 2       ║
║               2 ║ 16/09/2018              ║    -50 ║            19950 ║ 16/09/2018    ║ NULL        ║ 1       ║
║               7 ║ 16/09/2018              ║    -50 ║            19450 ║ 16/09/2018    ║ 28/10/2018  ║ 2       ║
║              12 ║ 16/09/2018              ║     50 ║            19950 ║ 16/09/2018    ║ 8/11/2018   ║ 3       ║
║               3 ║ 1/10/2018               ║    250 ║            20200 ║ 30/09/2018    ║ NULL        ║ 1       ║
║               8 ║ 1/10/2018               ║    250 ║            19700 ║ 30/09/2018    ║ 28/10/2018  ║ 2       ║
║              13 ║ 1/10/2018               ║    250 ║            20200 ║ 30/09/2018    ║ 8/11/2018   ║ 3       ║
║               4 ║ 6/10/2018               ║  -1200 ║            19000 ║ 6/10/2018     ║ NULL        ║ 1       ║
║               9 ║ 6/10/2018               ║  -1200 ║            17800 ║ 6/10/2018     ║ 28/10/2018  ║ 2       ║
║              14 ║ 6/10/2018               ║  -1200 ║            19000 ║ 6/10/2018     ║ 8/11/2018   ║ 3       ║
║               5 ║ 22/10/2018              ║    100 ║            19100 ║ 22/10/2018    ║ NULL        ║ 1       ║
║              10 ║ 22/10/2018              ║    100 ║            17900 ║ 22/10/2018    ║ 28/10/2018  ║ 2       ║
║              15 ║ 22/10/2018              ║    100 ║            19100 ║ 22/10/2018    ║ 8/11/2018   ║ 3       ║
║              11 ║ 29/10/2018              ║  -1000 ║            16900 ║ 29/10/2018    ║ NULL        ║ (New)1  ║
║              16 ║ 29/10/2018              ║  -1000 ║            18100 ║ 29/10/2018    ║ 8/11/2018   ║ (New)2  ║
╚═════════════════╩═════════════════════════╩════════╩══════════════════╩═══════════════╩═════════════╩═════════╝

Now when I run a report for September (2018-09-01 to 2018-09-30) I should be V1 or when SK_AsOfDate is NULL

If I run a report for October (2018-10-01 to 2018-10-31) my last record should be (11) with a principle balance of 16900

And my current Principle balance (As of 2018-11-09) should be calculated as of the balance from (16) with PB of (18100)

I have added the SK_AsOfDate to try and deal with versioning issue, but I am still struggling to see a simple and elegant way to achieve this "What was my balance as of 2018-09-30 that will ignore V2 and V3 alterations.

I want to get this right and luckily am not too far down any path, so suggestions are welcome! I am happy to add as many fields as makes this process simple for reporting out of the other end.

Answer 1

In the financial (and some other) transaction data you basically have two time dimensions .

Transaction date - representig the real time of the transaction happen, but due to technical reason you may recieve the transaction later.

Booking date - this is the timestamp the transaction entered your booking system. Sometimes called entry date .

Concerning the transaction date the transaction may occure as late arriving , contrary booking date is per definition always up to date.

The two time dimension allow two different kinds of reports .

The booking date report is typically used for bookkeping purposes (as it never changes in history). The transaction date report is more time realistic, but running it on two days for the last month can produce two different results (due to late transaction arrived on the second day).

Answer 2

It looks that is some problem with late arriving facts. The question is what you would like to report later. Would you like to report the new values somehow or just to ignore the new arriving facts.

The first step is to determine businesskey that would enable to notify the difference. Is the Amount or PrincipleBalancefor Key_Transaction changing in time, or just new records arriving? Try to create snapshots of table to find the diffenrence of values in time in order to create good busineskey.

Some good ideas can be found here: http://www.disoln.org/2013/12/Design-Approach-to-Handle-Late-Arriving-Dimensions-and-Late-Arriving-Facts.html

What is the source database ? In Sql Server you can try to use Change Data Capture (it has to be enabled on server) or create mechanism mentioned above in your ETL.

I guess that the table you mentioned is not a low level grain but some kind of already aggregation based on other tables. Try to ask what stays technically behind it and dig deeper to find how it works.

Answer 3

I think your case can be solved with "snapshot" tables. In financial world like you elaborated as of 2018-10-31 or as of 2018-11-09 is important and you need to keep a copy of your data for each "as of", it might be different for each org looks like yours is weekly. It is upto you to decide the frequency. When you have this set of data regardless of the final state you can go back and get an accurate report.

The way to create these "snapshot" tables is basically create a copy of your fact table on each "as of" date with a "snapshot date", this snapshot date aka "as of" could be used in your reports to see the version of data you need to see.

Let me know if this solves your problem.