在 SQL Server 2005/2008 中存储历史数据的最佳方式是什么？

Question

My simplified and contrived example is the following:-

Lets say that I want to measure and store the temperature (and other values) of all the worlds' towns on a daily basis. I am looking for an optimal way of storing the data so that it is just as easy to get the current temperature in all the towns, as it is to get all the temperature historically in one town.

It is an easy enough problem to solve, but I am looking for the best solution.

The 2 main options I can think of are as follows:-

Option 1 - Same table stores current and historical records

Store all the current and archive records in the same table.

ie

CREATE TABLE [dbo].[WeatherMeasurement](
  MeasurementID [int] Identity(1,1) NOT Null,
  TownID [int] Not Null,
  Temp [int] NOT Null,
  Date [datetime] NOT Null,
)

This would keep everything simple, but what would be the most efficient query to get a list of towns and there current temperature? Would this scale once the table has millions of rows in? Is there anything to be gained by having some sort of IsCurrent flag in the table?

Option 2 - Store all archive records in a separate table

There would be a table to store the current live measurements in

CREATE TABLE [dbo].[WeatherMeasurement](
  MeasurementID [int] Identity(1,1) NOT Null,
  TownID [int] Not Null,
  Temp [int] NOT Null,
  Date [datetime] NOT Null,
)

And a table to store historical archived date (inserted by a trigger perhaps)

CREATE TABLE [dbo].[WeatherMeasurementHistory](
  MeasurementID [int] Identity(1,1) NOT Null,
  TownID [int] Not Null,
  Temp [int] NOT Null,
  Date [datetime] NOT Null,
)

This has the advantages of keeping the main current data lean, and very efficient to query, at the expense of making the schema more complex and inserting data more expensive.

Which is the best option? Are there better options I haven't mentioned?

NOTE: I have simplified the schema to help focus my question better, but assume there will be alot of data inserted each day (100,000s of records), and data is current for one day. The current data is just as likely to be queried as the historical.

Answer 1

它取决于应用程序的使用模式......如果使用模式表明历史数据将比当前值更频繁地查询，那么将它们全部放在一个表中......但是如果历史查询是例外，（或少于10% 的查询），并且更常见的当前值查询的性能将受到将所有数据放在一个表中的影响，那么将这些数据分离到它自己的表中是有意义的......

Answer 2

I would keep the data in one table unless you have a very serious bias for current data (in usage) or history data (in volume). A compound index with DATE + TOWNID (in that order) would remove the performance concern in most cases (although clearly we don't have the data to be sure of this at this time).

The one thing I would wonder about is if anyone will want data from both the current and history data for a town. If so, you just created at least one new view to worry about and possible performance problem in that direction.

This is unfortunately one of those things where you may need to profile your solutions against real world data. I personally have used compound indexes such as specified above in many cases, and yet there are a few edge cases where I have opted to break the history into another table. Well, actually another data file, because the problem was that the history was so dense that I created a new data file for it alone to avoid bloating the entire primary data file set. Performance issues are rarely solved by theory.

I would recommend reading up on query hints for index use, and "covering indexes" for more information about performance issues.

Answer 3

Your table is very narrow and would probably perform in a single properly indexed table which would never outstrip the capacity of SQL Server in a traditional normalized OLTP model, even for millions and millions of rows. Even with dual-table model advantages can be mitigated by using table partitioning in SQL Server. So it doesn't have much to recommend it over the single table model. This would be an Inmon-style or "Enterprise Data Warehouse"- scenario.

In much bigger scenarios, I would transfer the data to a data warehouse (modeled with a Kimball-style dimensional model) on a regular basis and simply purge the live data - in some simple scenarios like yours, there might effectively be NO live data - it all goes straight into the warehouse. The dimensional model has a lot of advantages when slicing data different ways and storing huge numbers of facts with a variety of dimensions. Even in the data warehouse scenario, often fact tables are partitioned by date.

It might not seem like your data has this (Town and Date are your only explicit dimensions), however, in most data warehouses, dimensions can snowflake or there can be redundancy, so there would be other dimensions about the fact stored at time of load instead of snowflaking for more efficiency - like State, Zip Code, WasItRaining, IsStationUrban (contrived).

This might seem silly, but when you start to mine the data for results in data warehouses, this makes asking questions like - on a day with rain in urban environments, what was the average temperature in Maine? - just that little bit easier to get at without joining a whole bunch of tables (ie it doesn't require a lot of expertise on your normalized model and performs very quickly). Kind of like useless stats in baseball - but some apparently turn out to be useful.

Answer 4

I suggest keep in the same table since historical data is queried just as often. Unless you will be adding many more columns to the table.

When size becomes an issue, you can partition it out by decade and have a stored procedure union the requested rows.

Answer 5

Another alternative could be to go for one table for all data and have a view for the current temperature. This will not help performance but could well aid readability/maintainability. You could even go for an indexed view to improve performance if you have the appropriate version of sql .

Answer 6

I would use a single table with index views to provide me with the latest information. SQL 2005 and 2008 server are designed for data warehousing so should preform well under this condition.

If you have a data pattern that requires writing to the db often, then the best choice would be to have an active table and archive table that you batch update at some interval.

Answer 7

If you store all in one table how are you going to make a relational database.

Example:

id--------------GUID----PK

record_id-------GUID

every time a new record will be inserted the [id] will change but [record_id] will remain same. Now if you have to link it with address table how are you going to do that?

Answer 8

Instead of trying to optimize relational databases for this, you might want to consider using a Time series database . These are already optimized for dealing with time-based data. Some of their advantages are:

• Faster at querying time-based keys
• Large data throughput
  • Since default operation is just an append, this can be done very quickly. ( InfluxDb supports millions of data points per second ).
• Able to compress data more agressively
• More user-friendly for time-series data.
  • The API's tend to reflect typical use-cases for time-series data
  • Aggregate metrics can be automatically calculated (eg windowed averages)
  • Specific visualization tools are often available.

Personally I liked using the open source database InfluxDB , but other good alternatives are available.