ValueError: insecure string pickle

Question

When I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and

Answer 1

Check this thread. Peter Otten says:

A corrupted pickle. The error is raised if a string in the dump does not both start and end with " or '.

and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).

Answer 2

This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):

Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: insecure string pickle

This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.

Answer 3

This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:

Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.

The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.

Answer 4

Same problem with a file that was made with python on windows, and reloaded with python on linux. Solution : dos2unix on the file before reading in linux : works as a charm !