is python capable of running on multiple cores?

Question

Question: Because of python\'s use of \"GIL\" is python capable running its separate threads simultaneously?

---

Info:

After reading this I came away rathe

Answer 1

I reproduce the code above and in parallel it was quicker.

range(10000)
Sequential run time: 228.56 seconds
Parallel run time  : 147.03 seconds

my hardware and software setup:
Raspberry Pi 3 B+
Raspbian v 4.19.57-v7+ #1244
Python v. 3.5.3
GCC v. 6.3

During sequential process the CPU goes to 25% of utilization and during parallel process the CPU goes to 100%

Update on Jan/2020; today I did a new test. This time using my laptop I ran the code above:

range(1000000)
Sequential run time: 36.698254108428955 seconds
Parallel run time  : 25.329885959625244 seconds

my hardware and software setup:
W10
i7-8565-U
16 MB RAM
SSD 256MB

Python 3.8.1
MSC v.1916 64 bit (AMD64)

Answer 2

Python threads cannot take advantage of many cores. This is due to an internal implementation detail called the GIL (global interpreter lock) in the C implementation of python (cPython) which is almost certainly what you use.

The workaround is the multiprocessing module http://www.python.org/dev/peps/pep-0371/ which was developed for this purpose.

Documentation: http://docs.python.org/library/multiprocessing.html

(Or use a parallel language.)

Answer 3

example code taking all 4 cores on my ubuntu 14.04, python 2.7 64 bit.

import time
import threading


def t():
    with open('/dev/urandom') as f:
        for x in xrange(100):
            f.read(4 * 65535)

if __name__ == '__main__':
    start_time = time.time()
    t()
    t()
    t()
    t()
    print "Sequential run time: %.2f seconds" % (time.time() - start_time)

    start_time = time.time()
    t1 = threading.Thread(target=t)
    t2 = threading.Thread(target=t)
    t3 = threading.Thread(target=t)
    t4 = threading.Thread(target=t)
    t1.start()
    t2.start()
    t3.start()
    t4.start()
    t1.join()
    t2.join()
    t3.join()
    t4.join()
    print "Parallel run time: %.2f seconds" % (time.time() - start_time)

result:

$ python 1.py
Sequential run time: 3.69 seconds
Parallel run time: 4.82 seconds

Answer 4

I converted the script to Python3 and ran it on my Raspberry Pi 3B+:

import time
import threading

def t():
        with open('/dev/urandom', 'rb') as f:
                for x in range(100):
                        f.read(4 * 65535)

if __name__ == '__main__':
    start_time = time.time()
    t()
    t()
    t()
    t()
    print("Sequential run time: %.2f seconds" % (time.time() - start_time))

    start_time = time.time()
    t1 = threading.Thread(target=t)
    t2 = threading.Thread(target=t)
    t3 = threading.Thread(target=t)
    t4 = threading.Thread(target=t)
    t1.start()
    t2.start()
    t3.start()
    t4.start()
    t1.join()
    t2.join()
    t3.join()
    t4.join()
    print("Parallel run time: %.2f seconds" % (time.time() - start_time))

python3 t.py

Sequential run time: 2.10 seconds
Parallel run time: 1.41 seconds

For me, running parallel was quicker.

Answer 5

The answer is "Yes, But..."

But cPython cannot when you are using regular threads for concurrency.

You can either use something like multiprocessing, celery or mpi4py to split the parallel work into another process;

Or you can use something like Jython or IronPython to use an alternative interpreter that doesn't have a GIL.

A softer solution is to use libraries that don't run afoul of the GIL for heavy CPU tasks, for instance numpy can do the heavy lifting while not retaining the GIL, so other python threads can proceed. You can also use the ctypes library in this way.

If you are not doing CPU bound work, you can ignore the GIL issue entirely (kind of) since python won't aquire the GIL while it's waiting for IO.

Answer 6

CPython (the classic and prevalent implementation of Python) can't have more than one thread executing Python bytecode at the same time. This means compute-bound programs will only use one core. I/O operations and computing happening inside C extensions (such as numpy) can operate simultaneously.

Other implementation of Python (such as Jython or PyPy) may behave differently, I'm less clear on their details.

The usual recommendation is to use many processes rather than many threads.