NumPy arrays with SQLite

Question

The most common SQLite interface I\'ve seen in Python is sqlite3, but is there anything that works well with NumPy arrays or recarrays? By that I mean one that reco

Answer 1

Doug's suggestion with redis is quite good, but I think his code is a bit complicated and, as a result, rather slow. For my purposes, I had to serialize+write and then grab+deserialize a square matrix of about a million floats in less than a tenth of a second, so I did this:

For writing:

snapshot = np.random.randn(1024,1024)
serialized = snapshot.tobytes()
rs.set('snapshot_key', serialized)

Then for reads:

s = rs.get('snapshot_key')
deserialized = np.frombuffer(s).astype(np.float32)
rank = np.sqrt(deserialized.size).astype(int)
snap = deserialized(rank, rank)

You can do some basic performance testing with ipython using %time, but neither the tobytes or frombuffer take more than a few milliseconds.