Python - “can't pickle thread.lock” error when creating a thread under a multiprocess in Windows

问题

I'm getting stuck on what I think is a basic multiprocess and threading issue. I've got a multiprocess set up, and within this a thread. However, when I set up the thread class within the init function, I get the following error:

"TypeError: can't pickle thread.lock objects".

However, this does not happen if the thread is set up outside of the init function. Does anyone know why this is happening? Note I'm using Windows.

Some code is below to illustrate the issue. As typed below, it runs fine. However if print_hello() is called from within the DoStuff init def, then the error occurs, if it's called within the multi-process run() def then it's fine.

Can anyone point me in the right direction so it runs fine when called from init? thanks!

import multiprocessing
import threading
import time

class MyProcess(multiprocessing.Process):

    def __init__(self, **kwargs):
        super(MyProcess, self).__init__(**kwargs)
        self.dostuff = DoStuff()

    def run(self):
        print("starting DoStuff")
        # This works fine if the line below is uncommented and __init__ self.print_hello() is commented...
        self.dostuff.print_hello()


class DoStuff(object):
    def __init__(self, **kwargs):
        super(DoStuff, self).__init__(**kwargs)

        # If the following is uncommented, the error occurs...
        #   Note it also occurs if the lines in start_thead are pasted here...
        # self.print_hello()

    def print_hello(self):
        print "hello"
        self.start_thread()

    def start_thread(self):
        self.my_thread_instance = MyThread()
        self.my_thread_instance.start()
        time.sleep(0.1)


class MyThread(threading.Thread):
    def __init__(self):
        super(MyThread, self).__init__()

    def run(self):
        print("Starting MyThread")


if __name__ == '__main__':
    mp_target = MyProcess()       # Also pass the pipe to transfer data
    # mp_target.daemon = True
    mp_target.start()
    time.sleep(0.1)

回答1:

It looks like there is no simple answer, and it appears to be a restriction of Windows (Win 7, python 3.6 in my case); on Windows it looks like you need to start the process before you can start the worker thread inside the owned object.

There appears to be no such restriction on Unix (CentOS 7, python 2.7.5).

As an experiment I modified your code as follows; this version checks the OS and starts either the process first, or the thread first:

import multiprocessing
import threading
import time
import os

class MyProcess(multiprocessing.Process):

    def __init__(self, **kwargs):
        super(MyProcess, self).__init__(**kwargs)
        self.dostuff = DoStuff(self)

    def run(self):
        print("MyProcess.run()")
        print("MyProcess.ident = " + repr(self.ident))
        if os.name == 'nt':
            self.dostuff.start_thread()

class DoStuff(object):
    def __init__(self, owner, **kwargs):
        super(DoStuff, self).__init__(**kwargs)
        self.owner = owner
        if os.name != 'nt':
            self.start_thread()

    def start_thread(self):
        print("DoStuff.start_thread()")
        self.my_thread_instance = MyThread(self)
        self.my_thread_instance.start()
        time.sleep(0.1)

class MyThread(threading.Thread):
    def __init__(self, owner):
        super(MyThread, self).__init__()
        self.owner = owner

    def run(self):
        print("MyThread.run()")
        print("MyThread.ident = " + repr(self.ident))
        print("MyThread.owner.owner.ident = " + repr(self.owner.owner.ident))

if __name__ == '__main__':
    mp_target = MyProcess()       # Also pass the pipe to transfer data
    mp_target.daemon = True
    mp_target.start()
    time.sleep(0.1)

... and got the following on Windows, where the process starts first:

MyProcess.run()
MyProcess.ident = 14700
DoStuff.start_thread()
MyThread.run() 
MyThread.ident = 14220
MyThread.owner.owner.ident = 14700

... and the following on Linux, where the thread is started first:

DoStuff.start_thread()
MyThread.run()
MyThread.ident = 140316342347520
MyThread.owner.owner.ident = None
MyProcess.run()
MyProcess.ident = 4358

If it were my code I'd be tempted to always start the process first, then create the thread within that process; the following version works fine for me across both platforms:

import multiprocessing
import threading
import time

class MyProcess(multiprocessing.Process):

    def __init__(self, **kwargs):
        super(MyProcess, self).__init__(**kwargs)
        self.dostuff = DoStuff()

    def run(self):
        print("MyProcess.run()")
        self.dostuff.start_thread()

class DoStuff(object):
    def __init__(self, **kwargs):
        super(DoStuff, self).__init__(**kwargs)

    def start_thread(self):
        self.my_thread_instance = MyThread()
        self.my_thread_instance.start()
        time.sleep(0.1)

class MyThread(threading.Thread):
    def __init__(self):
        super(MyThread, self).__init__()

    def run(self):
        print("MyThread.run()")

if __name__ == '__main__':
    mp_target = MyProcess()       # Also pass the pipe to transfer data
    mp_target.daemon = True
    mp_target.start()
    time.sleep(0.1)

来源：https://stackoverflow.com/questions/44567402/python-cant-pickle-thread-lock-error-when-creating-a-thread-under-a-multipr