Most efficient way to append arrays in C#?

Question

I am pulling data out of an old-school ActiveX in the form of arrays of doubles. I don\'t initially know the final number of samples I will actually retrieve.

What i

Answer 1

You can't append to an actual array - the size of an array is fixed at creation time. Instead, use a List<T> which can grow as it needs to.

Alternatively, keep a list of arrays, and concatenate them all only when you've grabbed everything.

See Eric Lippert's blog post on arrays for more detail and insight than I could realistically provide :)

Answer 2

Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.

For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.

Answer 3

The solution looks like great fun, but it is possible to concatenate arrays in just two statements. When you're handling large byte arrays, I suppose it is inefficient to use a Linked List to contain each byte.

Here is a code sample for reading bytes from a stream and extending a byte array on the fly:

    byte[] buf = new byte[8192];
    byte[] result = new byte[0];
    int count = 0;
    do
    {
        count = resStream.Read(buf, 0, buf.Length);
        if (count != 0)
        {
            Array.Resize(ref result, result.Length + count);
            Array.Copy(buf, 0, result, result.Length - count, count);
        }
    }
    while (count > 0); // any more data to read?
    resStream.Close();

Answer 4

You might not need to concatenate end result into contiguous array. Instead, keep appending to the list as suggested by Jon. In the end you'll have a jagged array (well, almost rectangular in fact). When you need to access an element by index, use following indexing scheme:

double x = list[i / sampleSize][i % sampleSize];

Iteration over jagged array is also straightforward:

for (int iRow = 0; iRow < list.Length; ++iRow) {
  double[] row = list[iRow];
  for (int iCol = 0; iCol < row.Length; ++iCol) {
    double x = row[iCol];
  }
}

This saves you memory allocation and copying at expense of slightly slower element access. Whether this will be a net performance gain depends on size of your data, data access patterns and memory constraints.

Answer 5

Here is a usable class based on what Constantin said:

class Program
{
    static void Main(string[] args)
    {
        FastConcat<int> i = new FastConcat<int>();
        i.Add(new int[] { 0, 1, 2, 3, 4 });
        Console.WriteLine(i[0]);
        i.Add(new int[] { 5, 6, 7, 8, 9 });
        Console.WriteLine(i[4]);

        Console.WriteLine("Enumerator:");
        foreach (int val in i)
            Console.WriteLine(val);

        Console.ReadLine();
    }
}

class FastConcat<T> : IEnumerable<T>
{
    LinkedList<T[]> _items = new LinkedList<T[]>();
    int _count;

    public int Count
    {
        get
        {
            return _count;
        }
    }

    public void Add(T[] items)
    {
        if (items == null)
            return;
        if (items.Length == 0)
            return;

        _items.AddLast(items);
        _count += items.Length;
    }

    private T[] GetItemIndex(int realIndex, out int offset)
    {
        offset = 0; // Offset that needs to be applied to realIndex.
        int currentStart = 0; // Current index start.

        foreach (T[] items in _items)
        {
            currentStart += items.Length;
            if (currentStart > realIndex)
                return items;
            offset = currentStart;
        }
        return null;
    }

    public T this[int index]
    {
        get
        {
            int offset;
            T[] i = GetItemIndex(index, out offset);
            return i[index - offset];
        }
        set
        {
            int offset;
            T[] i = GetItemIndex(index, out offset);
            i[index - offset] = value;
        }
    }

    #region IEnumerable<T> Members

    public IEnumerator<T> GetEnumerator()
    {
        foreach (T[] items in _items)
            foreach (T item in items)
                yield return item;
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

Answer 6

If you can make an approximation of the number of items that will be there at the end, use the overload of the List constuctor that takes count as a parameter. You will save some expensive List duplications. Otherwise you have to pay for it.