pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han dle object at 0x00AAAAAAA>

问题

I'm trying to write a test to compare methods for inter process communication in python, and OSX and windows are giving me problem with my first sample. it is pretty much the manual sample for using Queues.

i wrote all the code on linux with python 2.7.9 and 3.4.1. And it works fine.

#!/usr/bin/env python
import multiprocessing as MP
...
self.mpq_main = MP.Queue()
self.mpq_net = MP.Queue()
self.mpt_net = MP.Process( target=self.start_t_net, args=(self.mpq_main, self.mpq_net) )
...
def start_t_net(self, qin, qout):
    self.NET = gcbtestnet.GCBTestNET(qin, qout);

Then testing on osx and windows i get errors. this question is for the windows one:

C:\Documents and Settings\user1\Desktop>C:\Python27\python.exe gcbtest.py
Traceback (most recent call last):
  File "gcbtest.py", line 82, in <module>
    theclass = GCBTest()
  File "gcbtest.py", line 31, in __init__
    self.mpt_fs.start();
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\multiprocessing\forking.py", line 67, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 725, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 313, in save
    (t.__name__, obj))
pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han
dle object at 0x00ABD190>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError
UI check
UI check
UI check
...

That error usually shows up if you try to pass a object reference as the process argument, but i am only passing multiprocess.Queues instances.

回答1:

I found my (gross) mistake. When i moved the Processes from simple methods to objects, i forgot to move the initialization of said objects to the .Process() call and kep it in a class method. For some reason it worked on linux, but not on windows.

here is the diff with the fix https://github.com/gcb/python_multiprocess_test/commit/8565384e5cc6cadf959c335b5db0693e646f6777

-self.mpt_net = MP.Process( target=self.start_t_net, args=(self.mpq_main, self.mpq_net) )
+self.mpt_net = MP.Process( target=gcbtestnet.GCBTestNET, args=(self.mpq_main, self.mpq_net) )
...
-def start_t_net(self, qin, qout):
-    self.NET = gcbtestnet.GCBTestNET(qin, qout);

来源：https://stackoverflow.com/questions/32003657/pickle-picklingerror-cant-pickle-subprocess-handle-object-subprocess-han