Delphi: Store data in somekind of structure

Question

For a simulation program I'm working in Delphi 2010. The simulation isn't a problem but I need to use large collection of data which gives a problem. The data is available in excel sheets, so there is no need to edit this data in Delphi, but collecting this data from the excel sheets takes around 10min. This isn't a problem as long as you don't need to collect the data every time the program runs. So I made a program which collects all the data makes it visible, not problems here,and then store it. However I can't store it to a "Delphi format" , without losing the structure, so it can be loaded in a few seconds.

I'm not that experienced in Delphi and I searched a long time for the solution but couldn't understand what was best. I think my way of structuring the data is wrong but it was simple and worked. However if there are better ways of storing the data please say so, but remember that I need some more explanation than just use 'a xml file', 'generict, or 'Ttreeview'. (have read it but wasn't able to use it).

The data is for: I made this product, The next product I make is this, so do I need to clean? True or false.

The data is stores as a class(TObject) with Productnumber (integer) and a List which contains all products that could be made next.This list contains another class(TObject) with an Productnumber (integer) and a do I need to clean(boolean). I want to save this structure in a file, without losing the data and read it back to the same structure.

I hope someone could help. Thank you in advance.

Update: The code to provide a little more information (modified to English)

Clean_from = class(TObject)
public
  myfromNumber      : Integer;
  mylist            : TList;
published
  constructor Create;
End

Clean_To = class(TObject)
public
  myToNumber        : Integer;
  Clean             : Boolean;
End;

constructor Clean_from.Create;
begin
  inherited Create;
  myList := Tlist.Create;
end;

For i = 0 to 100 do
begin
  From:= Clean_from.create;
  for j := 0 to 10 do 
  begin 
    To := Clean_To.create;
    To.clean := true or false;
    From.myList.add(To);
  end;
  GlobalList.add(from);
end;

And now I want to save the global list with all the content so I could load it with the same structure.

Answer 1

What you need is the so-called "serialization" mechanism.

1. The standard way

1.1 SaveToStream

In Delphi, we usually implement a SaveToStream method, which will save the content of each object in a destination TStream (either a TFileStream or a TMemoryStream).

You'll have to write the serialization by hand.

1.2 DFM-like streaming

See TWriter / TReader classes.

If you define your data in published properties, you are able to serialize them using those standard Delphi classes.

For some methods able to serialize any TCollection to and from JSON content, see this blog article.

2. The RTTI

See for instance this SO question.

In particular, the new enhanced RTTI (available since Delphi 2010) opens new opportunities to serialization.

3. Use records instead of classes

If each item does not store a lot of content (some integer/boolean), it may make sense to use records instead of objects. For speed and memory consumption/fragmentation, it may be worth it.

Here is some wrapper able to serialize any dynamic array, even containing nested records or dynamic arrays.

4. Use a database engine

Perhaps the better approach is not to have your data stuck in a non-evolving binary form, proprietary to your application. If you want to add a property, you'll have to manage it by hand. Or if you want to access your data from other applications, it may be difficult.

There are a lot of database solutions around - instead of using an external database (like MS SQL, FireBird or Oracle), it could be a good idea to embed the database inside your application (much easier to install). Worth mentioning SQLite which has a lot of wrappers, including our version (which will allow you to change to any other database if you want to use MS SQL or Oracle instead).

You have other solutions around - see this SO question - and if you need performance, take a look at our Big Table library.

Answer 2

Add SaveToStream() and LoadFromStream() methods to your data object which, well, save the data to a stream and load data from a stream.

type
  TMyData = class(TObject)
  private
    FChildProducts: TList;
    FProductnumber : integer;
    FClean: boolean;
  public
    procedure LoadFromStream(const aStream: TStream);
    procedure SaveToStream(const aStream: TStream);
  published
    property Productnumber: Integer read FProductnumber write FProductnumber;
    property Clean: Boolean reas FClean write FClean;
  end;

procedure TMyData.LoadFromStream(const aStream: TStream);
var x, cnt: Integer;
    cD: TMyData;
begin
  aStream.Read(FProductnumber, SizeOf(FProductnumber));
  aStream.Read(FClean, SizeOf(FClean));
  // read number of child products
  aStream.Read(cnt, SizeOf(cnt));
  // load child objects
  for x := 1 to cnt do begin
     cD := TMyData.create;
     cD.LoadFromStream(aStream);
     FChildProducts.Add(cD);
  end; 
end;

procedure TMyData.SaveToStream(const aStream: TStream);
var x: Integer;
begin
  aStream.Write(FProductnumber, SizeOf(FProductnumber));
  aStream.Write(FClean, SizeOf(FClean));
  // save number of child products
  x := FChildProducts.Count;
  aStream.Write(x, SizeOf(x));
  // save child objects
  for x := 0 to FChildProducts.Count - 1 do
     (FChildProducts[x] as TMyData).SaveToStream(aStream);
end;

I assume you have some list of "root objects" so you can make an function or method which saves/loads them to/from stream ie

function SaveDataList(const List: TList;const aFileName: string);
var x: Integer;
    FS: TFileStream;
begin
  FS := TFileStream.Create(aFileName, ...);
  try
     // save file version
     x := 1;
     FS.Write(x, SizeOf(x));
     // save number of products
     x := List.Count;
     FS.Write(x, SizeOf(x));
     // save objects
     for x := 0 to List.Count - 1 do
       (List[x] as TMyData).SaveToStream(FS);
  finally
     FS.Free;
  end;
end;

This is the general idea... how to load data back should be clear too. The file version thing is there so that when the data object changes (ie you add some property) you can increment the version number so that in the loading code you can load data into right version of the data object.

Answer 3

I would go for Arnoud's option #4, but use the ClientDataSet with nested ClientDataSets. This would allow you to load and save the data in a flexable structure as well as display it quickly. Check this page and also here for some nested dataset information and examples.