How do I fix this unicode/cPickle error in Python?

Question

ids = cPickle.loads(gem.value)

loads() argument 1 must be string, not unicode

Answer 1

The result of cPickle.dumps() is a str object, not a unicode object. You need to find the step in your code where you are decoding the pickled str object, and omit that step.

DON'T try to convert your unicode object to a str object. Two wrongs don't make a right. Example (Python 2.6):

>>> import cPickle
>>> ps = cPickle.dumps([1,2,3], -1)
>>> ps
'\x80\x02]q\x01(K\x01K\x02K\x03e.'
>>> ups = ps.decode('latin1')
>>> str(ups)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\x80' in position 0: ordinal not in range(128)
>>>

You may well be using the default (and inefficient) Protocol 0 which produces "human readable" output:

>>> ps = cPickle.dumps([1,2,3])
>>> ps
'(lp1\nI1\naI2\naI3\na.'
>>>

which is presumably ASCII (but not documented to be so) so the str(gem.value) kludge may well """work""":

>>> ps == str(unicode(ps))
True
>>>

Answer 2

cPickle.loads wants a byte string (which is exactly what cPickle.dumps outputs) and you're feeding it a unicode string instead. You'll need to "encode" that Unicode string to get back the byte string that dumps had originally given you, but it's hard to guess what encoding you accidentally imposed on it -- maybe latin-1 or utf-8 (if ascii don't worry, either of those two will decode it just great), maybe utf-16...? It's hard to guess without knowing what gem is and how you originally set its value from the output of a cPickle.dumps...!

Answer 3

You can fix it by making gem.value a string, not unicode.

Use str(gem.value)