Versioning in SQL Tables - how to handle it?

Question

Here\'s a fictional scenario with some populated data. For tax purposes, my fictional company must retain records of historical data. For this reason, I\'ve included a version c

Answer 1

You are definitely doing this wrong. Keeping a database running sweetly requires that you only have the minimum amount of data in your production tables that you need. Inevitably holding historical data in with the live adds redundancy that will complicate queries and slow performance, plus your successors are going to look really askew at this before submitting it to the DailyWTF!

Instead create a copy of the table - EmployeeHistorical for instance - but with the ID column not set as identity (you might choose to add an additional new ID column and a dateCreated timestamp column too). Then add a trigger to your Employee table that fires on update & delete and writes out a copy of the complete row to the Historical table. And while you're at it capturing the ID of the user doing the edit often comes in handy for audit purposes.

Generally when I'm doing this on an active table I try and create the historical table in a different database as among other things this reduces fragmentation (and hence maintenance) on your prime database and it's easier to handle backups - as archives can grow very large.

Your issues about edit contention should be handled with the normal database transaction and locking mechanisms. Coding adhoc hacks up to emulate such yourself is always time-consuming and error prone (some edge condition you've not thought of always pops up, and to write locks correctly you've really got to grok sempahores, which is decidedly non-trivial)

Answer 2

Here is my suggested approach, which has worked very well for me in the past:

• Forget the version number. Instead, use StartDate and EndDate columns
• Write a trigger to ensure that there are no overlapping date ranges for the same ID, and that there is only ever one record with a NULL EndDate for the same ID (this is your currently effective record)
• Put indexes on StartDate and EndDate; this should give you reasonable performance

This will easily let you report by date:

select *
from MyTable 
where MyReportDate between StartDate and EndDate

or get the current info:

select *
from MyTable 
where EndDate is null

Answer 3

An approach that I've designed for a recent database is to use revisions as follows:

• Keep your entity in two tables:

  1. "employee" stores a primary key ID and any data that you do not want to be versioned (if there is any).

  2. "employee_revision" stores all the salient data about the employee, with a foreign key to the employee table and a foreign key, "RevisionID" to a table called "revision".

• Make a new table called "revision". This can be used by all the entities in your database, not just employee. It contains an identity column for the primary key (or AutoNumber, or whatever your database calls such a thing). It also contains EffectiveFrom and EffectiveTo columns. I also have a text column on the table - entity_type - for human readability reasons which contain the name of the primary revision table (in this case "employee"). The revision table contains no foreign keys. The default value for EffectiveFrom is 1-Jan-1900 and the default value for EffectiveTo is 31-Dec-9999. This allows me to not simplify the date querying.

I make sure that the revision table is well indexed on (EffectiveFrom, EffectiveTo, RevisionID) and also on (RevisionID, EffectiveFrom, EffectiveTo).

I can then use joins and simple <> comparisons to select an appropriate record for any date. This also means that relations between entities are also fully versioned. In fact, I find it useful to use SQL Server table-valued functions to allow very simply querying of any date.

Here's an example (assuming that you don't want to version employee names so that if they change their name, the change is effective historically).

--------
employee
--------
employee_id  |  employee_name
-----------  |  -------------
12351        |  John Smith

-----------------
employee_revision
-----------------
employee_id  |  revision_id  |  department_id  |  position_id  |  pay
-----------  |  -----------  |  -------------  |  -----------  |  ----------
12351        |  657442       |  72             |  23           |  22000.00
12351        |  657512       |  72             |  27           |  22000.00
12351        |  657983       |  72             |  27           |  28000.00

--------
revision
--------
revision_id  |  effective_from  |  effective_to  |  entity_type
-----------  |  --------------  |  ------------  |  -----------
657442       |  01-Jan-1900     |  03-Mar-2007   |  EMPLOYEE
657512       |  04-Mar-2007     |  22-Jun-2009   |  EMPLOYEE
657983       |  23-Jun-2009     |  31-Dec-9999   |  EMPLOYEE

One advantage of storing your revision metadata in a separate table is that it's easy to apply it consistently to all your entities. Another is that it's easier to expand it to include other things, such as branches or scenarios, without having to modify every table. My principal reason is that it keeps your main entity tables clear and uncluttered.

(The data and example above are fictional - my database does not model employees).

Answer 4

Although the question has been asked 8 years ago, it worths to mention there is feature exactly for this in SQL Server 2016. System-versioned Temporal Table

Every table in SQL Server 2016 and above can have a history table, which the historical data will be populated automatically by SQL Server itself.

All you need is to add two datetime2 columns and one clause to the table:

CREATE TABLE Employee 
(
    Id int NOT NULL PRIMARY KEY CLUSTERED,
    [Name] varchar(50) NOT NULL,
    Position varchar(50)  NULL,
    Pay money NULL,
    ValidFrom datetime2 GENERATED ALWAYS AS ROW START NOT NULL,
    ValidTo datetime2 GENERATED ALWAYS AS ROW END NOT NULL,
        PERIOD FOR SYSTEM_TIME (ValidFrom,ValidTo)
)  
WITH (SYSTEM_VERSIONING = ON);

The system versioned table creates a temporal table which maintains the history of the data. You can use a custom name WITH (SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.EmployeeHistory ) );

In this link you can find more details about System-version temporal tables.

As @NotMe mentioned, historical tables can be grow very fast, so there are a few ways to get around this. Take a look here

Answer 5

Idea 3 will work:

SELECT * FROM EMPLOYEE AS e1
WHERE Position = 'Coder'
AND Version = (
    SELECT MAX(Version) FROM Employee AS e2
    WHERE e1.ID=e2.ID)

You really want to use something like a date though, which is much easier to program and track, and will use the same logic (something like an EffectiveDate column)

EDIT:

Chris is totally correct about moving this info out of your production table for performance, especially if you expect frequent updates. Another option would be to make a VIEW that only shows you the most recent version of each person's info, that you build off of this table.

Answer 6

What you have here is called a Slowly Changing Dimension (SCD). There are some proven methods for dealing with it:

http://en.wikipedia.org/wiki/Slowly_changing_dimension

Thought I'd add that since no one seems to call it by name.