C# - Parallelizing While Loop with StreamReader causing High CPU

问题

SemaphoreSlim sm = new SemaphoreSlim(10);

using (FileStream fileStream = File.OpenRead("..."))
using (StreamReader streamReader = new StreamReader(fileStream, Encoding.UTF8, true, 4096))
{
    String line;
    while ((line = streamReader.ReadLine()) != null)
    {
        sm.Wait();
        new Thread(() =>
        {
            doSomething(line);
            sm.Release();
        }).Start();
    }
}
MessageBox.Show("This should only show once doSomething() has done its LAST line.");

So, I have an extremely large file that I want to execute code on every single line.

I want to do it in Parallel but at a maximum of 10 at a time.

My solution for that was to use SemaphoreSlim to wait and release when the thread is finished. (Since the function is synchronous the placement of .Release() works).

The issue is the code takes a LOT of CPU. Memory is going just as expected and instead of loading in over 400mb, it just goes up and down a few mbs every few seconds.

But CPU goes crazy, its most of the time locked at 100% for a good 30 seconds and dips down slightly and goes back.

Since I don't want to load every line into memory, and want to run code as it goes, whats the best solution here?

500 Lines In on a 9,700 line file.

600 Lines In on a 2.7 million line file.

EDIT

I changed from new Thread(()=>{}).Start(); to Task.Factory.StartNew(()=>{}); as per mentioned in comments, it seems that the Thread Creation and Destroying is causing the performance drop. And it seems to be right. After I moved to Task.Factory.StartNew it runs same speed as per mentioned by the Semaphore, and its CPU is exactly like my Parallel.ForEach code version.

回答1:

Your code creates a huge number of threads, which is inefficient. C# has easier ways of handling with your scenario. One approach is:

File.ReadLines(path, Encoding.UTF8)
    .AsParallel().WithDegreeOfParallelism(10)
    .ForAll(doSomething);

File.ReadLines does not read the whole file, it reads line by line.
Use WithDegreeOfParallelism to set the maximal number of concurrent executing tasks
Use ForAll to start a method on each line.

来源：https://stackoverflow.com/questions/49217299/c-sharp-parallelizing-while-loop-with-streamreader-causing-high-cpu

标签

while-loop

semaphore

filestream

streamreader