Data structure/algorithm for variable length record storage and lookup on disk withsearch only on primary keys

Question

I am looking for an algorithm / data structure that works well for large block based devices (eg a mechanical hard drive) which is optimised for insert, get, update and delete w

Answer 1

Indexing variable-length-record files.

Indexing variable-length record files may look at first like a daunting task but it is really pretty straightforward once you identify which are the moving parts.

To do this you should read your file in blocks of fixed size (ie. 128, 256 or 512 etc). Your records also should have an easily identifiable end-of-record character. Meaning this character cannot appear as a regular character inside your records.

Next thing is you scan your file looking for the beginning of each record creating an index file with the following structure:

key, 0, 0
........
........
key, block, offset

Here key is the key (field) you are indexing your file on (can be a composite one). block is the (0-based) block number the record starts at, and offset is the (0-based) offset of the beginning of the record from the beginning of the block. Depending of the block size you use, your records may span more than one block. So once you locate the beginning of a record you need to fetch as many consecutive blocks as necessary to retrieve the whole record.

It is perfectly possible to create multiple index files at the same time if you need to search for different criteria.

Once you have created this index file, the next step is to sort it by the key field. Alternatively you can device an insertion sort mechanism which keeps your index file sorted as it is been created.

Retrieve your record sorted by that key using a file seek-like function, looking up the key on the index file and retrieving its record-offset pair. Binary search seems to perform pretty well in this scenario but you can use any type.

This structure allows your database to accept record additions, deletions and updates. Additions are made at the end of the file, adding its key to the index file. To delete a record, just change the first character of the record with a unique character like 0x0 and delete the entry from the index file. Updates can be achieved by deleting and then adding the updated record at the end of the file.

If you plan to deal with very large files then using a B-Tree like structure for your indexes may be of a great help as B-Trees indexes do not need to be loaded complete in memory. The B-Tree algorithm further divides the index file into pages which are then loaded in memory as needed.