Performance issues with nested loops and string concatenations

Question

Can someone please explain why this code is taking so long to run (i.e. >24 hours): The number of rows is 5000, whilst the number of columns is 2000 (i.e. Approximately 10m loop

Answer 1

Supposing that textToWrite is a String, you should use StringBuilder instead. String is immutable and it is very ineffective to add small parts.

Ideally you would initialize StringBuilder with a reasonable size (see doc).

Answer 2

Your code is taking so long because you're appending strings, creating thousands of new temporary strings as you go. The memory manager needs to find memory for these strings (which increase in memory requirements, as they get longer) and the operation copies the characters you have so far (the number of which increases with every iteration) to the newest string.

The alternative is to use a single StringBuilder, on which you call Append() to append more efficiently and, finally, ToString() when you're done to get the finalized string that you want to use.

Answer 3

Because you are creating tons of strings.

You should use StringBuilder for this.

StringBuilder sb = new StringBuildeR();

for (int i = 0; i < m.rows; i++)
{
    bool first = true;

    for (int j = 0; j < m.cols; j++)
    {
        sb.Append(m[i, j]);

        if (first)
        {
            first = false;
        }
        else
        {
            sb.Append(",");
        }
    }

    sb.AppendLine();
}

string output = sb.ToString();

Answer 4

Use a StringBuilder instead of several million concatenations.

If you concatenate 2 strings, this means the system allocates new memory to contain both of them, and then copies both in. A zillion large memory allocations and copy actions become slow very fast.

What StringBuilder does is reduce this immensely by allocating 'in advance', thus only having to grow the buffer a few times and just copying it in, eliminating the by far slowest factor of your loop.

Answer 5

Assume the matrix is of size MxM and has N elements. You are building the string in a way that takes O(N^2) (or O(M^4)) in the number of iterations. Each operation must copy what's already there. The issue is not some constant-factor overhead like temporary strings.

Use StringBuilder.

String concatenation is more efficient for small number of concatenated strings. For a dynamic number of strings, use StringBuilder.

Answer 6

The reason that it takes so long to run is because you are using string concatenation to create a string. For each iteration it will copy the entire string to a new string, so in the end you will have copied strings that adds up to several million times the final string.

Use a StringBuilder to create the string:

StringBuilder textToWrite = new StringBuilder();
for (int i = 0; i < m.rows; i++)
{
    for (int j = 0; j < m.cols; j++)
    {
        if (j > 0) textToWrite.Append(',');
        textToWrite.Append(m[i, j]);
    }
    textToWrite.AppendLine();
}