Python: How do I write a list to file and then pull it back into memory (dict represented as a string convert to dict) later?

Question

More specific dupe of 875228—Simple data storing in Python.

I have a rather large dict (6 GB) and I need to do some processing on it. I\'m trying out several docume

Answer 1

Here are a few alternatives depending on your requirements:

• numpy stores your plain data in a compact form and performs group/mass operations well

• shelve is like a large dict backed up by a file

• some 3rd party storage module, e.g. stash, stores arbitrary plain data

• proper database, e.g. mongodb for hairy data or mysql or sqlite plain data and faster retrieval