Can't pickle defaultdict

Question

I have a defaultdict that looks like this:

dict1 = defaultdict(lambda: defaultdict(int))

The problem is, I can't pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?

Answer 1

In addition to Martijn's explanation:

A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it's not nested within another function, and it is a "real" function with a name, not a lambda function.

So, to pickle your defaultdict, create it with module-level function instead of a lambda function:

def dd():
    return defaultdict(int)

dict1 = defaultdict(dd) # dd is a module-level function

than you can pickle it

tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)

Answer 2

Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.

Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.

Answer 3

You can however use partial to accomplish this:

>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})

Answer 4

To do this, just write the code you wanted to write. I'd use dill, which can serialize lambdas and defaultdicts. Dill can serialize almost anything in python.

>>> import dill
>>> from collections import defaultdict
>>>
>>> dict1 = defaultdict(lambda: defaultdict(int))
>>> pdict1 = dill.dumps(dict1)
>>> _dict1 = dill.loads(pdict1)
>>> _dict1
defaultdict(<function <lambda> at 0x10b31b398>, {})

Answer 5

If you don't care about preserving the defaultdict type, convert it:

fname = "file.pkl"

for value in nested_default_dict:
    nested_default_dict[value] = dict(nested_default_dict[value])
my_dict = dict(nested_default_dict)

with open(fname, "wb") as f:
    pickle.dump(my_dict, f)  # Now this will work

I think this is a great alternative since when you are pickling, the object is probably in it's final form... AND, if really do need the defaultdict type again, you can simply convert is back after you unpickle:

for value in my_dict:
    my_dict[value] = defaultdict(type, my_dict[value])
nested_default_dict = defaultdict(type, my_dict)

Answer 6

dict1 = defaultdict(lambda: defaultdict(int))
cPickle.dump(dict(dict1), file_handle)

worked for me

Answer 7

I'm currently doing something similar to the question poser, however, I'm using a subclass of defaultdict which has a member function that is used as the default_factory. In order to have my code work properly (I required the function to be defined at runtime), I simply added some code to prepare the object for pickling.

Instead of:

...
pickle.dump(dict, file)
...

I use this:

....
factory = dict.default_factory
dict.default_factory = None
pickle.dump(dict, file)
dict.default_factory = factory
...

This isn't the exact code I used as my tree is an object which creates instances of the same the tree's type as indexes are requested (so I use a recursive member function to do the pre/post pickle operations), but this pattern also answers the question.