Pickle can't store an object in django locmem cache during tests?

Question

Something that\'s puzzling me a bit...

>>> from django.core.cache import get_cache
>>>
>>> cache = get_cache(\'django.core.cache.backe         


        

Answer 1

The problem is that pickle serializes classes by reference, so could you not just use a better serializer that pickles the class by serializing the class definitions instead of by reference? Then you'd pickle a mock object, which would then pickle the class source code, and then you'd be able to pass that to the django cache. I'm the author of dill, which is a better serializer… and also the author of klepto, which is a caching package… and this is exactly what I do to store any object in a SQL table, on disk, or in an in-memory cache.

Essentially (not trying this, but guessing it works based on experience with my own caching package), it should work like this:

>>> from django.core.cache import get_cache
>>> import dill
>>>
>>> cache = get_cache('django.core.cache.backends.locmem.LocMemCache')
>>>
>>> # Set the 'content' cache key to a string
>>> cache.set('content', dill.dumps('a string'))
>>> dill.loads(cache.get('content'))
'a string'
>>>
>>> class TestObj(object):
...     pass
>>>
>>> a = TestObj()
>>> cache.set('content', dill.dumps(a))
>>>
>>> dill.loads(cache.get('content'))
<__main__.TestObj object at 0x10235e510>
>>>
>>> # this is pickling classes w/o using a reference
>>> dill.dumps(a)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x07TestObjq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13b.'
>>> # and here's using a reference, which is exactly how pickle does it
>>> dill.dumps(a, byref=True)
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'

If you want to try it yourself, get dill (and klepto) here: https://github.com/uqfoundation

Answer 2

sample code

import pickle  # or from django.utils.six.moves import cPickle as pickle
lass TestObj(object):
     pass
testobj = TestObj()   
pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)
pickled
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'

i can't understand a behavior, when i open a console session with "python manage.py shell" and execute: _pickle.PicklingError: Can't pickle : attribute lookup builtins.TestObj failed

but when open a single python console and execute the same code, works fine! Is there a reason?

I observed too, if i open a single console, and import from django.utils.six.moves import cPickle as pickle works fine too. The problem just occours when the code is executed in django context. :(

Answer 3

WIth help from Martijn in this follow-up question it turns out the short answer is:

"Yup".

You can't pickle Mock() objects as they do not provide the top-level object that they are mocking so pickle therefore has no idea where to import from. As the cache requires the object to be pickled to store it, it is not possible to store a Mock() instance in the LocMemCache. Will have to rethink how I go about testing this.

Answer 4

It happens because django LocMemCache uses cPickle instead of pickle by default. You can see it in LocMemCache class:

try:
    from django.utils.six.moves import cPickle as pickle
except ImportError:
    import pickle

If you will try to do in shell:

from django.utils.six.moves import cPickle as pickle
testobj = TestObj()
pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)

It will be the same error.

As possible solution I propose you to pack objects manually in your tests using pickle and after that do cache.set():

a = TestObj()
pickled = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
cache.set('content', pickled)