Multithreading: What is the point of more threads than cores?

Question

I thought the point of a multi-core computer is that it could run multiple threads simultaneously. In that case, if you have a quad-core machine, what\'s the point of having

Answer 1

Many threads will be asleep, waiting for user input, I/O, and other events.

Answer 2

The point is that the vast majority of programmers do not understand how to design a state machine. Being able to put everything in its own thread frees the programmer from having to think about how to efficiently represent the state of different in-progress computations so that they can be interrupted and later resumed.

As an example, consider video compression, a very cpu-intensive task. If you're using a gui tool, you probably want the interface to remain responsive (show progress, respond to cancel requests, window resizing, etc.). So you design your encoder software to process a large unit (one or more frames) at a time and run it in its own thread, separate from the UI.

Of course once you realize it would have been nice to be able to save the in-progress encoding state so you can close the program to reboot or play a resource-hungry game, you realize you should have learned how to design state machines from the beginning. Either that, or you decide to engineer a whole new problem of process-hibernation your OS so you can suspend and resume individual apps to disk...

Answer 3

The answer revolves around the purpose of threads, which is parallelism: to run several separate lines of execution at once. In an 'ideal' system, you would have one thread executing per core: no interruption. In reality this isn't the case. Even if you have four cores and four working threads, your process and it threads will constantly be being switched out for other processes and threads. If you are running any modern OS, every process has at least one thread, and many have more. All these processes are running at once. You probably have several hundred threads all running on your machine right now. You won't ever get a situation where a thread runs without having time 'stolen' from it. (Well, you might if it's running real-time, if you're using a realtime OS or, even on Windows, use a real-time thread priority. But it's rare.)

With that as background, the answer: Yes, more than four threads on a true four-core machine may give you a situation where they 'steal time from each other', but only if each individual thread needs 100% CPU. If a thread is not working 100% (as a UI thread might not be, or a thread doing a small amount of work or waiting on something else) then another thread being scheduled is actually a good situation.

It's actually more complicated than that:

• What if you have five bits of work that all need to be done at once? It makes more sense to run them all at once, than to run four of them and then run the fifth later.

• It's rare for a thread to genuinely need 100% CPU. The moment it uses disk or network I/O, for example, it may be potentially spend time waiting doing nothing useful. This is a very common situation.

• If you have work that needs to be run, one common mechanism is to use a threadpool. It might seem to make sense to have the same number of threads as cores, yet the .Net threadpool has up to 250 threads available per processor. I'm not certain why they do this, but my guess is to do with the size of the tasks that are given to run on the threads.

So: stealing time isn't a bad thing (and isn't really theft, either: it's how the system is supposed to work.) Write your multithreaded programs based on the kind of work the threads will do, which may not be CPU-bound. Figure out the number of threads you need based on profiling and measurement. You may find it more useful to think in terms of tasks or jobs, rather than threads: write objects of work and give them to a pool to be run. Finally, unless your program is truly performance-critical, don't worry too much :)

Answer 4

The way some API are designed, you have no choice but to run them in a separate thread (anything with blocking operations). An example would be Python's HTTP libraries (AFAIK).

Usually this isn't much of a problem though (if it is a problem, the OS or API should ship with an alternative asynchronous operating mode, ie: select(2)), because it probably means the thread is going to be sleeping during waiting for I/O completion. On the other hand, if something is doing a heavy computation, you have to put it in a separate thread than say, the GUI thread (unless you enjoy manual multiplexing).

Answer 5

A thread is an abstraction that enables You to write code as simple as a sequence of operation, blissfully unaware of that the code is executed interlaced with other code, or parked waiting for IO, or (maybe somewhat more aware of) waiting for other thread's events or messages.

Answer 6

I strongly disagree with @kyoryu's assertion that the ideal number is one thread per CPU.

Think about it this way: why do we have multi-processing operating systems? For most of computer history, nearly all computers had one CPU. Yet from the 1960s on, all "real" computers had multi-processing (aka multi-tasking) operating systems.

You run multiple programs so that one can run while others are blocked for things like IO.

lets set aside arguments about whether Windows versions before NT were multi-tasking. Since then, every real OS had multi-tasking. Some don't expose it to users, but its there anyway, doing things like listening to the cellphone radio, talking to the GPS chip, accepting mouse input, etc.

Threads are just tasks that are a bit more efficient. There is no fundamental difference between a task, process, and thread.

A CPU is a terrible thing to waste, so have lots of things ready to use it when you can.

I will agree that with most procedural languages, C, C++, Java etc, writing proper thread safe code is a lot of work. With 6 core CPUs on the market today, and 16 core CPUs not far away, I expect that folks will move away from these old languages, as multi-threading is more and more of a critical requirement.

Disagreement with @kyoryu is just IMHO, the rest is fact.