How to add items to a collection while consuming it?

Question

The example below throws an InvalidOperationException, \"Collection was modified; enumeration operation may not execute.\" when executing the code.

var urls         


        

Answer 1

Consider using a Queue with while loop (while q.Count > 0, url = q.Dequeue()) instead of iteration.

Answer 2

Don't change the collection you're looping through via for each. Just use a while loop on the Count property of the list and access the List items by index. This way, even if you add items, the iteration should pick up the changes.

Edit: Then again, it sort of depends on whether you WANT the new items you added to be picked up by the loop. If not, then this won't help.

Edit 2: I guess the easiest way to do it would be to just change your loop to: foreach (string url in urls.ToArray())

This will create an Array copy of your list, and it will loop through this instead of the original list. This will have the effect of not looping over your added items.

Answer 3

I assume you want to iterate over the whole list, and each item you add to it? If so I would suggest recursion:

var urls = new List<string>();
var turls = new List<string();
turls.Add("http://www.google.com")

iterate(turls);

function iterate(List<string> u)
{
    foreach(string url in u)
    {
        List<string> newUrls = GetLinks(url);

        urls.AddRange(newUrls);

        iterate(newUrls);
    }
}

Answer 4

Jon's approach is right; a queue's the right data structure for this kind of application.

Assuming that you'd eventually like your program to terminate, I'd suggest two other things:

• don't use string for your URLs, use System.Web.Uri: it provides a canonical string representation of the URL. This will be useful for the second suggestion, which is...
• put the canonical string representation of each URL you process in a Dictionary. Before you enqueue a URL, check to see if it's in the Dictionary first.

Answer 5

There are three strategies you can use.

1. Copy the List<> to a second collection (list or array - perhaps use ToArray()). Loop through that second collection, adding urls to the first.
2. Create a second List<>, and loop through your urls List<> adding new values to the second list. Copy those to the original list when done looping.
3. Use a for loop instead of a foreach loop. Grab your count up front. List should leave things indexed correctly, so it you add things they will go to the end of the list.

I prefer #3 as it doesn't have any of the overhead associated with #1 or #2. Here is an example:

var urls = new List<string>();
urls.Add("http://www.google.com");
int count = urls.Count;

for (int index = 0; index < count; index++)
{
    // Get all links from the url
    List<string> newUrls = GetLinks(urls[index]);

    urls.AddRange(newUrls);
}

---

Edit: The last example (#3) assumes that you don't want to process additional URLs as they are found in the loop. If you do want to process additional URLs as they are found, just use urls.Count in the for loop instead of the local count variable as mentioned by configurator in the comments for this answer.

Answer 6

alternately, you could treat the collection as a queue

IList<string> urls = new List<string>();
urls.Add("http://www.google.com");
while (urls.Count > 0)
{
    string url = urls[0];
    urls.RemoveAt(0);
    // Get all links from the url
    List<string> newUrls = GetLinks(url);
    urls.AddRange(newUrls);
}