Why is this C++ code execution so slow compared to java?

Question

I recently wrote a computation-intensive algorithm in Java, and then translated it to C++. To my surprise the C++ executed considerably slower. I have now written a much shorter

Answer 1

Somehow both GCC and clang fail to unroll this loop and pull out the invariants even in -O3 and -Os, but Java does.

Java's final JITted assembly code is similar to this (in reality repeated twice):

    while (true) {
        loopCount++;
        if (++intArray[i++] >= FINISH_TRIGGER) break;
        loopCount++;
        if (++intArray[i++] >= FINISH_TRIGGER) break;
        loopCount++;
        if (++intArray[i++] >= FINISH_TRIGGER) break;
        loopCount++;
        if (++intArray[i++] >= FINISH_TRIGGER) { if (i >= ARRAY_LENGTH) i = 0; break; }
        if (i >= ARRAY_LENGTH) i = 0;
    }

With this loop I'm getting exact same timings (6.4s) between C++ and Java.

Why is this legal to do? Because ARRAY_LENGTH is 100, which is a multiple of 4. So i can only exceed 100 and be reset to 0 every 4 iterations.

This looks like an opportunity for improvement for GCC and clang; they fail to unroll loops for which the total number of iterations is unknown, but even if unrolling is forced, they fail to recognize parts of the loop that apply to only certain iterations.

Regarding your findings in a more complex code (a.k.a. real life): Java's optimizer is exceptionally good for small loops, a lot of thought has been put into that, but Java loses a lot of time on virtual calls and GC.

In the end it comes down to machine instructions running on a concrete architecture, whoever comes up with the best set, wins. Don't assume the compiler will "do the right thing", look and the generated code, profile, repeat.

For example, if you restructure your loop just a bit:

    while (!finished) {
        for (i=0; i<ARRAY_LENGTH; ++i) {
            loopCount++;
            if (++intArray[i] >= FINISH_TRIGGER) {
                finished=true;
                break;
            }
        }
    }

Then C++ will outperform Java (5.9s vs 6.4s). (revised C++ assembly)

And if you can allow a slight overrun (increment more intArray elements after reaching the exit condition):

    while (!finished) {
        for (int i=0; i<ARRAY_LENGTH; ++i) {
            ++intArray[i];
        }
        loopCount+=ARRAY_LENGTH;
        for (int i=0; i<ARRAY_LENGTH; ++i) {
            if (intArray[i] >= FINISH_TRIGGER) {
                loopCount-=ARRAY_LENGTH-i-1;
                finished=true;
                break;
            }
        }
    }

Now clang is able to vectorize the loop and reaches the speed of 3.5s vs. Java's 4.8s (GCC is unfortunately still not able to vectorize it).

Answer 2

The only time I could get the C++ program to outperform Java was when using profiling information. This shows that there's something in the runtime information (that Java gets by default) that allows for faster execution.

There's not much going on in your program apart from a non-trivial if statement. That is, without analysing the entire program, it's hard to predict which branch is most likely. This leads me to believe that this is a branch misprediction issue. Modern CPUs do instruction pipelining which allows for higher CPU throughput. However, this requires a prediction of what the next instructions to execute are. If the guess is wrong, the instruction pipeline must be cleared out, and the correct instructions loaded in (which takes time).

At compile time, the compiler doesn't have enough information to predict which branch is most likely. CPUs do a bit of branch prediction as well, but this is generally along the lines of loops loop and ifs if (rather than else).

Java, however, has the advantage of being able to use information at runtime as well as compile time. This allows Java to identify the middle branch as the one that occurs most frequently and so have this branch predicted for the pipeline.