What is the best way to use an embedded database, say sqlite in Python:
There's an easy-to-use Python module that meets all the stated objectives:
http://yserial.sourceforge.net/
Serialization + persistance: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data.
Surprisingly, there is not much difference between in-memory and persistent solutions for most practical purposes.
As for "shield me from the actual database", with y_serial, one cannot even tell that SQLite is behind it all. If you construct your records as Python dictionaries, you can concentrate on just writing code (not stored procedures).