High memory consumption with Enumerable.Range?

Question

Originally i wanted to know whether ToList allocates more memory than using the constructor of List which takes an IEnumerable

Answer 1

List is implemented as an array. When you exceed what it has allocated, it allocates another array double the size (essentially doubling the memory allocation). The default capacity is 4, and it doubles things from here on.

Most likely if you drop the number of items to say 7,500, you'll see the array drop to a little under 32 MBs, and the IList size to be 32 MBs.

You can tell IList<T> what the initial size should be, which is why if you give it the IEnumerable<T> at construction time, it shouldn't over allocate memory.

[Edit] after comments

In the case of Enumerable.Range(a, b) it returns an IEnumerable<T> only rather than an ICollection<T>. For List<T> to not overallocate the item passed during construction must also be an ICollection<T>

Answer 2

It's because of the doubling algorithm used to create the backing array in a List. IEnumerable has no Count property so it can't pre-allocate the backing array to be the target size when you call ToList. In fact, on each call to MoveNext you are calling a corresponding Add on the List.

However Array.ToList can override the base ToList behaviour to initialise the list to the correct capacity. Also, it could be the List in it's constructor which tries to downcast the IEnumerable reference it has to known collection types such as IList, ICollection, Array, etc...

Update

In fact it is in the constructor of List that it figures out if the argument implements ICollection:

public List(IEnumerable<T> collection)
{
  if (collection == null)
    ThrowHelper.ThrowArgumentNullException(ExceptionArgument.collection);
  ICollection<T> collection1 = collection as ICollection<T>;
  if (collection1 != null)
  {
    int count = collection1.Count;
    if (count == 0)
    {
      this._items = List<T>._emptyArray;
    }
    else
    {
      this._items = new T[count];
      collection1.CopyTo(this._items, 0);
      this._size = count;
    }
  }
  else
  {
    this._size = 0;
    this._items = List<T>._emptyArray;
    foreach (T obj in collection)
      this.Add(obj);
  }
}

Answer 3

This probably relates to the doubling algorithm used to resize the backing buffer when adding to a list. When you allocate as an array, the length of that is known, and can be queried by checking for IList[<T>] and/or ICollection[<T>]; thus it can allocate a single array, right-sized the first time, and then just block-copy the contents.

With the sequence this is not possible (the sequence does not expose the length in any accessible way); thus it must instead fall back to "keep filling up the buffer; if full, double it and copy".

Obviously this needs approx double the memory.

An interesting test would be:

var list = new List<int>(10000000);
list.AddRange(Enumerable.Range(1, 10000000));

This will allocate the right size initially, while still using the sequence.

tl;dr; the constructor, when passed a sequence, first checks to see if it can obtain the length by casting to a well-known interface.

Answer 4

I guess that:

• Enumerable.Range(1, 10000000) only creates an IEnumerable and doesn't create items yet.

• Enumerable.Range(1, 10000000).ToArray() creates an array, using memory for the numbers

• Enumerable.Range(1, 10000000).ToList() creates the numbers and additional data to manage the list (links between parts. The list can change its size and needs to allocate memory in blocks).