Why does my Python program average only 33% CPU per process? How can I make Python use all available CPU?

Question

I use Python 2.5.4. My computer: CPU AMD Phenom X3 720BE, Mainboard 780G, 4GB RAM, Windows 7 32 bit.

I use Python threading but can not make every python.exe process co

Answer 1

Global Interpreter Lock

The reasons of employing such a lock include:

* increased speed of single-threaded programs (no necessity to acquire or release locks
  on all data structures separately)
* easy integration of C libraries that usually are not thread-safe.

Applications written in languages with a GIL have to use separate processes (i.e. interpreters) to achieve full concurrency, as each interpreter has its own GIL.

Answer 2

From CPU usage it looks like you're still running on a single core. Try running a trivial calculation with 3 or more threads with same threading code and see if it utilizes all cores. If it doesn't, something might be wrong with your threading code.

Answer 3

You bottleneck is probably somewhere else, like the hard-drive (paging), or memory access.

Answer 4

What about Stackless Python?

Answer 5

You should perform some Operating System and Python monitoring to determine where the bottle neck is.

Here is some info for windows 7:

Performance Monitor: You can use Windows Performance Monitor to examine how programs you run affect your computer’s performance, both in real time and by collecting log data for later analysis. (Control Panel-> All Control Panel Items->Performance Information and Tools-> Advanced Tools- > View Performance Monitor)

Resource Monitor: Windows Resource Monitor is a system tool that allows you to view information about the use of hardware (CPU, memory, disk, and network) and software (file handles and modules) resources in real time. You can use Resource Monitor to start, stop, suspend, and resume processes and services. (Control Panel-> All Control Panel Items->Performance Information and Tools-> Advanced Tools- > View Resource Monitor)

Answer 6

It appears that you have a 3-core CPU. If you want to use more than one CPU core in native Python code, you have to spawn multiple processes. (Two or more Python threads cannot run concurrently on different CPUs)

As R. Pate said, Python's multiprocessing module is one way. However, I would suggest looking at Parallel Python instead. It takes care of distributing tasks and message-passing. You can even run tasks on many separate computers with little change to your code.

Using it is quite simple:

import pp

def parallel_function(arg):
    return arg

job_server = pp.Server() 

# Define your jobs
job1 = job_server.submit(parallel_function, ("foo",))
job2 = job_server.submit(parallel_function, ("bar",))

# Compute and retrieve answers for the jobs.
print job1()
print job2()