问题
I have a defaultdict that looks like this:
dict1 = defaultdict(lambda: defaultdict(int))
The problem is, I can\'t pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?
回答1:
In addition to Martijn's explanation:
A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it's not nested within another function, and it is a "real" function with a name, not a lambda function.
So, to pickle your defaultdict, create it with module-level function instead of a lambda function:
def dd():
return defaultdict(int)
dict1 = defaultdict(dd) # dd is a module-level function
than you can pickle it
tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)
回答2:
Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.
Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.
回答3:
You can however use partial to accomplish this:
>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})
回答4:
To do this, just write the code you wanted to write. I'd use dill, which can serialize lambdas and defaultdicts. Dill can serialize almost anything in python.
>>> import dill
>>> from collections import defaultdict
>>>
>>> dict1 = defaultdict(lambda: defaultdict(int))
>>> pdict1 = dill.dumps(dict1)
>>> _dict1 = dill.loads(pdict1)
>>> _dict1
defaultdict(<function <lambda> at 0x10b31b398>, {})
回答5:
dict1 = defaultdict(lambda: defaultdict(int))
cPickle.dump(dict(dict1), file_handle)
worked for me
回答6:
If you don't care about preserving the defaultdict type, convert it:
fname = "file.pkl"
for value in nested_default_dict:
nested_default_dict[value] = dict(nested_default_dict[value])
my_dict = dict(nested_default_dict)
with open(fname, "wb") as f:
pickle.dump(my_dict, f) # Now this will work
I think this is a great alternative since when you are pickling, the object is probably in it's final form... AND, if really do need the defaultdict type again, you can simply convert is back after you unpickle:
for value in my_dict:
my_dict[value] = defaultdict(type, my_dict[value])
nested_default_dict = defaultdict(type, my_dict)
回答7:
I'm currently doing something similar to the question poser, however, I'm using a subclass of defaultdict which has a member function that is used as the default_factory. In order to have my code work properly (I required the function to be defined at runtime), I simply added some code to prepare the object for pickling.
Instead of:
...
pickle.dump(dict, file)
...
I use this:
....
factory = dict.default_factory
dict.default_factory = None
pickle.dump(dict, file)
dict.default_factory = factory
...
This isn't the exact code I used as my tree is an object which creates instances of the same the tree's type as indexes are requested (so I use a recursive member function to do the pre/post pickle operations), but this pattern also answers the question.
回答8:
Implementing the anonlymous lambda function by normal function worked for me. As pointed out by Mike, Pickle cannot handle lambdas; pickle only handles data.Hence, converting my method from:
dict_ = defaultdict(lambda: default_value)
to:
default_():
return default_value
and then creating the default dict as follows worked for me:
dict_ = defaultdict(default_)
来源：https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict