Pickle dump huge file without memory error

Question

I have a program where I basically adjust the probability of certain things happening based on what is already known. My file of data is already saved as a pickle

Answer 1

Have you tried using streaming pickle: https://code.google.com/p/streaming-pickle/

I have just solved a similar memory error by switching to streaming pickle.