Copy the list you are iterating. Then remove from the copy and interate the original. Going backwards is confusing and doesn't work well when looping in parallel.
var ids = new List { 1, 2, 3, 4 };
var iterableIds = ids.ToList();
Parallel.ForEach(iterableIds, id =>
{
ids.Remove(id);
});