How to limit number of cores with threading

问题

The code posted below initiates a single Thread that fires up 4 cores on my macbookpro. Is there a way to limit to how many cores the thread should be using?

    import threading

    import logging
    logging.basicConfig(level=logging.DEBUG, format='(%(threadName)s) %(message)s',)

    def func():
        logging.debug('starting')
        m=0
        for i in range(500000000):
            m+=i
        logging.debug('exiting') 

    thread = threading.Thread(target=func)
    thread.start()

    logging.debug('completed')

Here is the log:

Loaded sitecustomize.py
(Thread-1) starting
(MainThread) completed
(Thread-1) exiting
[Finished in 128.0s]

回答1:

There is multithreading and multiprocessing.

You can bind a process to a cpu-core, but you can't bind a thread to a core.

The main difference between processes and threads are creation time, threads spawn faster and they run in the same memory space, while processes have separate memory.

import multiprocessing as mp
import psutil


def spawn():
    procs = list()
    n_cpus = psutil.cpu_count()
    for cpu in xrange(n_cpus):
        affinity = [cpu]
        d['affinity'] = affinity
        p = mp.Process(target=run_child, kwargs=d)
        p.start()
        procs.append(p)
    for p in procs:
        p.join()
        print('joined')

def run_child(affinity):
    proc = psutil.Process()  # get self pid
    print('PID: {pid}'.format(pid=proc.pid))
    aff = proc.cpu_affinity()
    print('Affinity before: {aff}'.format(aff=aff))
    proc.cpu_affinity(affinity)
    aff = proc.cpu_affinity()
    print('Affinity after: {aff}'.format(aff=aff))


if __init__ == '__main__':
    spawn()

回答2:

since you are using multithreading it is actually is a single process. so the problem is how to make that process pin to a cpu, i.e. How to set processor affinity on OS X? a lot of things are explained in this question, althou the symptom is not same as yours.

回答3:

What you're asking about is called "Processor Affinity". Normally, the operating system's scheduler takes care of this and it's something you shouldn't worry about. Yes, your thread is bouncing between 4 cores but it's only really using about 25% of each one.

Here's a SuperUser question about setting processor afinity in OSX. It doesn't look like there's anything to fiddle with it in the Python standard library since there's very few practical reasons to do so.

No, wanting your CPU usage charts to look prettier is not a valid reason to set CPU affinities.

回答4:

You can set process CPU affinity in Python with psutil:

>>> import psutil
>>> psutil.cpu_count()
4
>>> p = psutil.Process()
>>> p.cpu_affinity()  # get
[0, 1, 2, 3]
>>> p.cpu_affinity([0])  # set; from now on, process will run on CPU #0 only
>>> p.cpu_affinity()
[0]
>>>

来源：https://stackoverflow.com/questions/41401490/how-to-limit-number-of-cores-with-threading

标签

python

multithreading