How can C#'s string.IndexOf perform so fast, 10 times faster than ordinary for loop find?

Question

I have a very long string (60MB in size) in which I need to find how many pairs of \'<\' and \'>\' are in there.

---

I have first tried my own way:

Answer 1

The only thing that comes to my mind is actual implementation of IndexOf iniside string class, that call

callvirt    System.String.IndexOf

which, if we use a power of reflector (as much as it possible) ends up into the

CompareInfo.IndexOf(..)

call, which instead use super fast windows native function FindNLSString:

Answer 2

It's a bit of a fallacy to directly compare your managed implementation and the String.IndexOf method. The implementation of IndexOf is largely native code and hence has a different set of performance characteristics than your managed implementation. In particular native code avoid the type safety checks, and their corresponding overhead, which injected by the CLR in the managed algorithm.

One example is the use of the array index

char c = html[j];

In managed code this statement will both verify that j is a valid index into the array html and then return the value. The native code equivalent simply returns a memory offset with no additional checking necessary. This lack of a check gives native code an inherent advantage here because it can avoid an additional branch instruction on every iteration of the loop.

Note that this advantage is not absolute. The JIT could very well examine this loop and decide that j could never be an invalid index and elide the checks in the generated native code (and it does do this in some cases IIRC).

Answer 3

I would expect string.IndexOf compared to your first code sample to run at least twice as fast (besides any other optimizations that might be done there) since you check for both start and end character in each of your iterations. Your implementation with string.IndexOf on the other hand will check for the end character only after it has successfully found a start character. This cuts the number of operations on each iteration down significantly (one less comparison).

Answer 4

Are you running your timings from within Visual Studio? If so, your code would run significantly slower for that reason alone.

Aside from that, you are, to some degree, comparing apples and oranges. The two algorithms work in a different way.

The IndexOf version alternates between looking for an opening bracket only and a closing bracket only. Your code goes through the whole string and keeps a status flag that indicates whether it is looking for an opening or a closing bracket. This takes more work and is expected to be slower.

Here's some code that does the comparison the same way as your IndexOf method.

int match3 = 0;
for (int j = 0; j < html.Length; j++) {
    if (html[j] == '<') {
        for (; j < html.Length; j++)
            if (html[j] == '>')
                match3++;
    }
}

In my tests this is actually about 3 times faster than the IndexOf method. The reason? Strings are actually not quite as simple as sequences of individual characters. There are markers, accents, etc. String.IndexOf handles all that extra complexity properly, but it comes at a cost.

Answer 5

Using a switch statement instead of an if test speeds things up a little also. This code occasionally beats the indexof code on my machine.

        int count = 0;
        bool open = false;
        for (int j = 0; j < testStr.Length; j++)
        {  
            switch (testStr[j])
            {
                case '<':
                    open = true;
                    break;
                case '>':
                    if (open)
                       count++;

                    open = false;
                    break;         
            }
        }