First let me tell you that you're optimizing the wrong thing here: unless you're doing this for recreational purposes, then your approach is wrong. XML is not a difficult format but it does have it quirks and it takes it's liberties. It's a format designed for data exchange between foreign applications, so the emphasis needs to be put on COMPATIBILITY, not on SPEED! What good is a non-standard ultra-fast parser that gives the wrong result when confronted with a slightly altered XML file?
If you can find a XML parsing LIBRARY that's guaranteed to be compatible with anything out there that can parse your data at HALF the speed your HDD can read it, then simply implement a producer-consumer multi-threaded application where one thread constantly reads the data from disk while the other two simply do the parsing. In the end you'll only be limited by the speed of the HDD while maintaining compatibility. If you're only looking for speed you're liable to make mistakes, skip XML features, depend on certain particularities of the sample XML file you're dealing with. Your application is likely to break for numerous reasons.
Remember that the most costly cycle for an application is MAINTENANCE, not production. What you might gain today by making a 50% faster thingy that's 200% percent more difficult to maintain will be lost in a year or so, when computers get 50% faster (nulling your edge over the competition). Besides, there's no point in exceeding natural limits for such processes, like the speed of the HDD. It's irrelevant that you're testing with a file from a RAM-drive - when the application goes into production it will be used with files from a HDD, and your application's performance will be limited by the speed of your HDD.
Anyhow, I do like a challenge once in a while and I really like parsers. What follows is a very simple parser implementation that only looks at each character in the input string once and only copies stuff where needed: copies the name of the nodes in order to decide what to do next and copies the node's "Payload" when appropriate, in order to push it into the array. On my "modest" i7 @ 3.4 Ghz parsing a string built by copying your sample data 10,000 times takes 63 ms. It clearly beats your time, BUT a word of warning, this code is fragile: it depends on having a XML file that's a certain form. No way around that.
program Project28;
{$APPTYPE CONSOLE}
uses SysUtils, DateUtils, Windows;
const SampleData =
''#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
' randomdata'#13#10+
''#13#10;
const NodeIterations = 10000;
type
TDummyRecord = record
D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13,
D14, D15, D16, D17, D18, D19, D20: string;
end;
TDummyRecordArray = array[1..NodeIterations] of TDummyRecord;
procedure ParseDummyXMLToRecordArray(const InputText:string; var A: TDummyRecordArray);
var PInputText: PChar;
cPos, TextLen: Integer;
C: Char;
State: Integer;
tag_starts_at: Integer;
last_payload_starts_at: Integer;
FlagEndTag: Boolean;
NodeName, Payload: string;
cNode: Integer;
const st_not_in_node = 1;
st_in_node = 2;
begin
cPos := 1;
TextLen := Length(InputText);
PInputText := @InputText[1];
State := st_not_in_node;
last_payload_starts_at := 1;
cNode := 0;
// This is the lexer/parser loop. It's a finite-state machine with only
// two states: st_not_in_node and st_in_node
while cPos < TextLen do
begin
C := PInputText[cPos-1];
case State of
// What happens when we're NOT currently inside a node?
// Not much. We only jump to st_in_node if we see a "<"
st_not_in_node:
case C of
'<':
begin
// A node starts here. Switch state and set up some simple
// flags.
state := st_in_node;
tag_starts_at := cPos + 1;
FlagEndTag := False;
end;
end;
// What happens while inside a node? Again, not much. We only care about
// the "/" - as it signals an closing tag, and we only care about the
// ">" because that means the end of the ndoe.
st_in_node:
case C of
'/': FlagEndTag := True;
'>':
begin
// This is where the magic haepens. We're in one of possibly two states:
// We're ither seeing the first of a pair, or the second
//
if FlagEndTag then
begin
// This is the closing pair of a tag pair, ie, it's the What we'll do
// depends on what node is closing, so we retreive the NodeName:
NodeName := System.Copy(InputText, tag_starts_at+1, cPos - tag_starts_at-1);
if NodeName <> 'node' then // SAMPLE-DATA-SPECIFIC: I know I don't care about "node" tags.
begin
// SAMPLE-DATA-SPECIFIC: I know there are only two kinds of nodes:
// "node" and "datatypeN". I retreive the PAYLOAD for the node because
// I know it's not "ndoe" and I know I'll need it.
Payload := System.Copy(InputText,last_payload_starts_at, tag_starts_at - last_payload_starts_at -1);
// Make sure we're dealing with a valid node
if (cNode > 0) and (cNode <= High(A)) then
begin
// Based on NodeName, copy the Payload into the appropriate field.
if NodeName = 'datatype1' then A[cNode].D1 := Payload
else if NodeName = 'datatype2' then A[cNode].D2 := Payload
else if NodeName = 'datatype3' then A[cNode].D3 := Payload
else if NodeName = 'datatype4' then A[cNode].D4 := Payload
else if NodeName = 'datatype5' then A[cNode].D5 := Payload
else if NodeName = 'datatype6' then A[cNode].D6 := Payload
else if NodeName = 'datatype7' then A[cNode].D7 := Payload
else if NodeName = 'datatype8' then A[cNode].D8 := Payload
else if NodeName = 'datatype9' then A[cNode].D9 := Payload
else if NodeName = 'datatype10' then A[cNode].D10 := Payload
else if NodeName = 'datatype11' then A[cNode].D11 := Payload
else if NodeName = 'datatype12' then A[cNode].D12 := Payload
else if NodeName = 'datatype13' then A[cNode].D13 := Payload
else if NodeName = 'datatype14' then A[cNode].D14 := Payload
else if NodeName = 'datatype15' then A[cNode].D15 := Payload
else if NodeName = 'datatype16' then A[cNode].D16 := Payload
else if NodeName = 'datatype17' then A[cNode].D17 := Payload
else if NodeName = 'datatype18' then A[cNode].D18 := Payload
else if NodeName = 'datatype19' then A[cNode].D19 := Payload
else if NodeName = 'datatype20' then A[cNode].D20 := Payload
else
raise Exception.Create('Unknown node: ' + NodeName);
end
else
raise Exception.Create('cNode out of bounds.');
end;
// Repeat :-)
state := st_not_in_node;
end
else
begin
// Node start. Retreive node name. I only care about the start of the "NODE" - if I see that
// I'll increment the current node counter so I'll go on filling the next position in the array
// with whatever I need.
NodeName := System.Copy(InputText, tag_starts_at, cPos - tag_starts_at);
last_payload_starts_at := cPos+1;
if NodeName = 'node' then Inc(cNode);
state := st_not_in_node;
end;
end;
end;
end;
Inc(cPos);
end;
end;
var DataString: string;
SB: TStringBuilder;
i: Integer;
DummyArray: TDummyRecordArray;
T1, T2, F: Int64;
begin
try
try
// Prepare the sample string; 10.000 iterations of the sample data.
SB := TStringBuilder.Create;
try
for i:=1 to NodeIterations do
SB.Append(SampleData);
DataString := SB.ToString;
finally SB.Free;
end;
// Invoke the simple parser using the string constant.
QueryPerformanceCounter(T1);
ParseDummyXMLToRecordArray(DataString, DummyArray);
QueryPerformanceCounter(T2);
QueryPerformanceFrequency(F);
WriteLn(((T2-T1) * 1000) div F);
// Test parse validity.
for i:=1 to NodeIterations do
begin
if DummyArray[i].D1 <> 'randomdata' then raise Exception.Create('Bug. D1 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D2 <> 'randomdata' then raise Exception.Create('Bug. D2 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D3 <> 'randomdata' then raise Exception.Create('Bug. D3 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D4 <> 'randomdata' then raise Exception.Create('Bug. D4 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D5 <> 'randomdata' then raise Exception.Create('Bug. D5 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D6 <> 'randomdata' then raise Exception.Create('Bug. D6 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D7 <> 'randomdata' then raise Exception.Create('Bug. D7 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D8 <> 'randomdata' then raise Exception.Create('Bug. D8 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D9 <> 'randomdata' then raise Exception.Create('Bug. D9 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D10 <> 'randomdata' then raise Exception.Create('Bug. D10 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D11 <> 'randomdata' then raise Exception.Create('Bug. D11 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D12 <> 'randomdata' then raise Exception.Create('Bug. D12 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D13 <> 'randomdata' then raise Exception.Create('Bug. D13 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D14 <> 'randomdata' then raise Exception.Create('Bug. D14 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D15 <> 'randomdata' then raise Exception.Create('Bug. D15 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D16 <> 'randomdata' then raise Exception.Create('Bug. D16 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D17 <> 'randomdata' then raise Exception.Create('Bug. D17 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D18 <> 'randomdata' then raise Exception.Create('Bug. D18 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D19 <> 'randomdata' then raise Exception.Create('Bug. D19 doesn''t have the proper value, i=' + IntToStr(i));
if DummyArray[i].D20 <> 'randomdata' then raise Exception.Create('Bug. D20 doesn''t have the proper value, i=' + IntToStr(i));
end;
except on E: Exception do Writeln(E.ClassName, ': ', E.Message);
end;
finally
WriteLn('ENTER to Exit');
ReadLn;
end;
end.
