Delphi: Store data in somekind of structure

Question

For a simulation program I\'m working in Delphi 2010. The simulation isn\'t a problem but I need to use large collection of data which gives a problem. The data is available

Answer 1

Add SaveToStream() and LoadFromStream() methods to your data object which, well, save the data to a stream and load data from a stream.

type
  TMyData = class(TObject)
  private
    FChildProducts: TList;
    FProductnumber : integer;
    FClean: boolean;
  public
    procedure LoadFromStream(const aStream: TStream);
    procedure SaveToStream(const aStream: TStream);
  published
    property Productnumber: Integer read FProductnumber write FProductnumber;
    property Clean: Boolean reas FClean write FClean;
  end;

procedure TMyData.LoadFromStream(const aStream: TStream);
var x, cnt: Integer;
    cD: TMyData;
begin
  aStream.Read(FProductnumber, SizeOf(FProductnumber));
  aStream.Read(FClean, SizeOf(FClean));
  // read number of child products
  aStream.Read(cnt, SizeOf(cnt));
  // load child objects
  for x := 1 to cnt do begin
     cD := TMyData.create;
     cD.LoadFromStream(aStream);
     FChildProducts.Add(cD);
  end; 
end;

procedure TMyData.SaveToStream(const aStream: TStream);
var x: Integer;
begin
  aStream.Write(FProductnumber, SizeOf(FProductnumber));
  aStream.Write(FClean, SizeOf(FClean));
  // save number of child products
  x := FChildProducts.Count;
  aStream.Write(x, SizeOf(x));
  // save child objects
  for x := 0 to FChildProducts.Count - 1 do
     (FChildProducts[x] as TMyData).SaveToStream(aStream);
end;

I assume you have some list of "root objects" so you can make an function or method which saves/loads them to/from stream ie

function SaveDataList(const List: TList;const aFileName: string);
var x: Integer;
    FS: TFileStream;
begin
  FS := TFileStream.Create(aFileName, ...);
  try
     // save file version
     x := 1;
     FS.Write(x, SizeOf(x));
     // save number of products
     x := List.Count;
     FS.Write(x, SizeOf(x));
     // save objects
     for x := 0 to List.Count - 1 do
       (List[x] as TMyData).SaveToStream(FS);
  finally
     FS.Free;
  end;
end;

This is the general idea... how to load data back should be clear too. The file version thing is there so that when the data object changes (ie you add some property) you can increment the version number so that in the loading code you can load data into right version of the data object.

Answer 2

What you need is the so-called "serialization" mechanism.

1. The standard way

1.1 SaveToStream

In Delphi, we usually implement a SaveToStream method, which will save the content of each object in a destination TStream (either a TFileStream or a TMemoryStream).

You'll have to write the serialization by hand.

1.2 DFM-like streaming

See TWriter / TReader classes.

If you define your data in published properties, you are able to serialize them using those standard Delphi classes.

For some methods able to serialize any TCollection to and from JSON content, see this blog article.

2. The RTTI

See for instance this SO question.

In particular, the new enhanced RTTI (available since Delphi 2010) opens new opportunities to serialization.

3. Use records instead of classes

If each item does not store a lot of content (some integer/boolean), it may make sense to use records instead of objects. For speed and memory consumption/fragmentation, it may be worth it.

Here is some wrapper able to serialize any dynamic array, even containing nested records or dynamic arrays.

4. Use a database engine

Perhaps the better approach is not to have your data stuck in a non-evolving binary form, proprietary to your application. If you want to add a property, you'll have to manage it by hand. Or if you want to access your data from other applications, it may be difficult.

There are a lot of database solutions around - instead of using an external database (like MS SQL, FireBird or Oracle), it could be a good idea to embed the database inside your application (much easier to install). Worth mentioning SQLite which has a lot of wrappers, including our version (which will allow you to change to any other database if you want to use MS SQL or Oracle instead).

You have other solutions around - see this SO question - and if you need performance, take a look at our Big Table library.