What is the best way to store historical data in SQL Server 2005/2008?

Question

My simplified and contrived example is the following:-

Lets say that I want to measure and store the temperature (and other values) of all the worlds\' towns on a daily

Answer 1

Your table is very narrow and would probably perform in a single properly indexed table which would never outstrip the capacity of SQL Server in a traditional normalized OLTP model, even for millions and millions of rows. Even with dual-table model advantages can be mitigated by using table partitioning in SQL Server. So it doesn't have much to recommend it over the single table model. This would be an Inmon-style or "Enterprise Data Warehouse"- scenario.

In much bigger scenarios, I would transfer the data to a data warehouse (modeled with a Kimball-style dimensional model) on a regular basis and simply purge the live data - in some simple scenarios like yours, there might effectively be NO live data - it all goes straight into the warehouse. The dimensional model has a lot of advantages when slicing data different ways and storing huge numbers of facts with a variety of dimensions. Even in the data warehouse scenario, often fact tables are partitioned by date.

It might not seem like your data has this (Town and Date are your only explicit dimensions), however, in most data warehouses, dimensions can snowflake or there can be redundancy, so there would be other dimensions about the fact stored at time of load instead of snowflaking for more efficiency - like State, Zip Code, WasItRaining, IsStationUrban (contrived).

This might seem silly, but when you start to mine the data for results in data warehouses, this makes asking questions like - on a day with rain in urban environments, what was the average temperature in Maine? - just that little bit easier to get at without joining a whole bunch of tables (i.e. it doesn't require a lot of expertise on your normalized model and performs very quickly). Kind of like useless stats in baseball - but some apparently turn out to be useful.