ValueError: insecure string pickle

Question

When I am trying to load something I dumped using cPickle, I get the error message:

ValueError: insecure string pickle

Both the dumping and

Answer 1

I got the Python ValueError: insecure string pickle message in a different way.

For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.

Initially I was wrapping up a file like this

with open(path_to_binary_file) as data_file:
    contents = data_file.read() 
filename = os.path.split(path)[1]

url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()

server_response = urllib2.urlopen(url, base64_message)

But on the server the hash kept coming out differently for some binary files

decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()

And unpickling gave insecure string pickle message

cPickle.loads(decoded_message)

BUT SUCCESS

What worked for me was to use urlsafe_b64encode()

base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))

And decode with

base64_decoded_message = base64.urlsafe_b64decode(base64_message)

References

http://docs.python.org/2/library/base64.html

https://tools.ietf.org/html/rfc3548.html#section-3

Answer 2

What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.

Answer 3

"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.

One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: insecure string pickle

Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:

>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))

Easy to forget. Didn't need people being told that they are idiots.

Answer 4

I've get this error in Python 2.7 because of open mode 'rb':

    with open(path_to_file, 'rb') as pickle_file:
        obj = pickle.load(pickle_file)

So, for Python 2 'mode' should be 'r'

Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:

pickle.unpicklingerror: the string opcode argument must be quoted

Answer 5

If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695

richiecannizzo commented on Aug 28

brew install libav
Should fix it instantly on mac or

sudo apt-get install libav
#on linux

Answer 6

I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)

    # Push file to another machine
    _send_file(temp_file.name)

However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.

Two potential fixes to this: Keep the file open and force a flush:

with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
    pickle.dump(foo, temp_file)
    temp_file.flush()

    # Push file to another machine
    _send_file(temp_file.name)

Or make sure the file is closed before doing anything with it:

file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
    file_name = temp_file.name
    pickle.dump(foo, temp_file)

# Push file to another machine
_send_file(file_name)