Pickle class definition in module with dill

问题

My module contains a class which should be pickleable, both instance and definition I have the following structure:

MyModule
|-Submodule
  |-MyClass

In other questions on SO I have already found that dill is able to pickle class definitions and surely enough it works by copying the definition of MyClass into a separate script and pickling it there, like this:

import dill as pickle

class MyClass(object):
    ...

instance = MyClass(...)
with open(..., 'wb') as file:
   pickle.dump(instance, file)

However, it does not work when importing the class:

Pickling:

from MyModule.Submodule import MyClass
import dill as pickle

instance = MyClass(...)
with open(.., 'wb') as file:
    pickle.dump(instance, file)

Loading:

import dill as pickle

with open(..., 'rb') as file:
    instance = pickle.load(file)

>>> ModuleNotFoundError: No module named 'MyModule'

I think the class definition is saved by reference, although it should not have as per default settings in dill. This is done correctly when MyClass is known as __main__.MyClass, which happens when the class is defined in the main script.

I am wondering, is there any way to detach MyClass from MyModule? Any way to make it act like a top level import (__main__.MyClass) so dill knows how to load it on my other machine?

Relevant question: Why dill dumps external classes by reference no matter what

回答1:

I'm the dill author. This is a duplicate of the question you refer to above. The relevant GitHub feature request is: https://github.com/uqfoundation/dill/issues/128.

I think the larger issue is that you want to pickle an object defined in another file that is not installed. This is currently not possible, I believe.

As a workaround, I believe you may be able to pickle with dill.source by extracting the source code of the class (or module) and pickling that dynamically, or extracting the source code and compiling a new object in __main__.

回答2:

I managed to save the instance and definition of my class using the following dirty hack:

class MyClass(object):
    def save(path):
        import __main__

        with open(__file__) as f:
            code = compile(f.read(), "somefile.py", 'exec')
            globals = __main__.__dict__
            locals = {'instance': self, 'savepath': path}
            exec(code, globals, locals)

if __name__ == '__main__':
    # Script is loaded in top level, MyClass is now available under the qualname '__main__.MyClass'
    import dill as pickle

    # copy the attributes of the 'MyModule.Submodule.MyClass' instance to a bew 'MyClass' instance.
    new_instance = MyClass.__new__(MyClass)
    new_instance.__dict__ = locals()['instance'].__dict__

    with open(locals()['savepath'], 'wb') as f:       
        pickle.dump(new_instance, f)

Using the exec statement the file can be executed from within __main__, so the class definition will be saved as well. This script should not be executed as main script without using the save function.

来源：https://stackoverflow.com/questions/52402783/pickle-class-definition-in-module-with-dill

标签

python

python-3.x

serialization

pickle

dill