Gap-less sequence where multiple transactions with multiple tables are involved

Question

I have a requirement (by law) for a gap-less numbers on different tables. The IDs can have holes in them but not the sequences.

This is something I have to either so

Answer 1

As you have already seemed to conclude, gapless sequences simply do not scale. Either you run the risk of dropping values when a rollback occurs, or you have a serialization point that will prevent a multi-user, concurrent transaction system from scaling. You cannot have both.

My thought would be, what about a post processing action, where every day, you have a process that runs at close of business, checks for gaps, and renumbers anything that needs to be renumbered?

One final thought: I don't know your requirement, but, I know you said this is "required by law". Well, ask yourself, what did people do before there were computers? How would this "requirement" be met? Assuming you have a stack of blank forms that come preprinted with a "sequence" number in the upper right corner? And what happens if someone spilled coffee on that form? How was that handled? It seems you need a similar method to handle that in your system.

Hope that helps.

Answer 2

Here is an idea that should support both high performance and high concurrency:

1. Use a highly concurrent, cached Oracle sequence to generate a dumb unique identifier for the gap-less table row. Call this entity MASTER_TABLE

2. Use the dumb unique identifier for all internal referential integrity from the MASTER_TABLE to other dependent detail tables.

3. Now your gap-less MASTER_TABLE sequence number can be implemented as an additional attribute on the MASTER_TABLE, and will be populated by a process that is separate from the MASTER_TABLE row creation. In fact, the gap-less additional attribute should be maintained in a 4th normal form attribute table of the MASTER_TABLE, and hence then a single background thread can then populate it at leisure, without concern for any row-locks on the MASTER_TABLE.

4. All queries that need to display the gap-less sequence number on a screen or report or whatever, would join the MASTER_TABLE with the gap-less additional attribute 4th normal form table. Note, these joins will be satisfied only after the background thread had populated the gap-less additional attribute 4th normal form table.

Answer 3

Gap-less sequences are hard to come by. I suggest to use a plain serial column instead. Create a view with the window function row_number() to produce a gap-less sequence:

CREATE VIEW foo AS
SELECT *, row_number() OVER (ORDER BY serial_col) AS gapless_id
FROM   tbl;

Answer 4

This problem is impossible to solve by principle because any transaction can rollback (bugs, timeouts, deadlocks, network errors, ...).

You will have a serial contention point. Try to reduce contention as much as possible: Keep the transaction that is allocating numbers as small as possible. Also, allocate numbers as late as possible in the transaction because only once you allocate a number contention arises. if you're doing 1000ms of uncontended work, and then allocate a number (taking 10ms) you still have a degree of parallelism of 100 which is enough.

So maybe you can insert all rows (of which you say there are many) with dummy sequence numbers, and only at the end of the transaction you quickly allocate all real sequence numbers and update the rows that are already written. This would work well if there are more inserts than updates, or the updates are quicker than the inserts (which they will be), or there is other processing or waiting interleaved between the inserts.