I have something similar to this in my code:
Parallel.ForEach(myList, new ParallelOptions { MaxDegreeOfParallelism = 4 }, item => { Process(item); });
Consider adding CancellationToken to your code. This way, at any point you can properly cancel all the operations.
CancellationToken
Then, you can use the CancelAfter() method.
CancelAfter()