When to use threading and how many threads to use

Question

I have a project for work. We had written a module and there as a #TODO to implement threading to improve the module. I\'m a fairly new python programmer and decided to take

Answer 1

No, this is not a good place to use threads.

Generally, you want to use threads where your code is IO-bound; that is, it spends a significant amount of time waiting on input or output. An example might be downloading data from a list of URLs in parallel; the code can start requesting the data from the next URL while still waiting for the previous one to return.

That's not the case here; calculating primes is cpu-bound.

Answer 2

You're right to think that multithreading is a questionable move here for good reason. Multithreading, as it stands, is great and in the right applications can make a worlds difference in running times.

However, on the other hand, it also adds additional complexity to any program that implements it (especially in python). There are also time penalties to consider when using multithreading, such as those that occur when doing context switches or the time it takes to actually create a thread.

These time penalties are negligent when your program has to process thousands upon thousands of resource intense tasks because the time you would save from having multithreading far outweighs the little bit of time it takes to get the threads ready. For your case though, I'm not sure your needs meet those requirements. I didn't look to deep into what type of objects you were processing but you stated they only took about 2 seconds, which isn't awful and you also said that you only have 6 items at a time to process. So on average we can expect the main part of your scrip to run for 12 seconds. In my opinion, that is not necessary for multithreading because it will take a second or two to get the threads ready and then pass the instructions to them, whereas in one thread your python script would already be well into processing its second object in that time.

In short, I would save multithreading unless you need it. For example, huge datasets like those used for gene sequencing (big thing in Python) benefit greatly from it because multiple threads can help process these massive files concurrently. In your case, it doesn't look like the ends justify the means. Hope this helps

Answer 3

Threading in python is used to run multiple threads (tasks, function calls) at the same time. Note that this does not mean that they are executed on different CPUs. Python threads will NOT make your program faster if it already uses 100 % CPU time. In that case, you probably want to look into parallel programming.

from: https://en.wikibooks.org/wiki/Python_Programming/Threading

This is due to the mechanism called GIL. As Daniel pointed out, threads in python are only useful when you have IO-bound code. But then again, for IO-bound code it may be better to use lighter threads running on top of some event loop (using gevent, eventlet, asyncio or similar) as then you can easily run 100s (and more) of parallel operations with very little per thread overhead.

If what you want is to use more than 1 core of CPU to speed up execution, take a look at multiprocessing module.