Is it possible to store Python class objects in SQLite?

Question

I would like to store Python objects into a SQLite database. Is that possible?

If so what would be some links / examples for it?

Answer 1

Yes it's possible but there are different approaches and which one is the suitable one, will depend on your requirements.

• Pickling

  You can use the pickle module to serialize objects, then store these objects in a blob in sqlite3 (or a textfield, if the dump is e.g. base64 encoded). Be aware of some possible problems: questions/198692/can-i-pickle-a-python-dictionary-into-a-sqlite3-text-field

• Object-Relational-Mapping

  You can use object relational mapping. This creates, in effect, a "virtual object database" that can be used from within the programming language (Wikipedia). For python, there is a nice toolkit for that: sqlalchemy.

Answer 2

You can use pickle.dumps, its return pickable objects as strings, you would not need to write it to temporary files.

Return the pickled representation of the object as a string, instead of writing it to a file.

import pickle

class Foo:
    attr = 'a class attr'

picklestring = pickle.dumps(Foo)

Answer 3

You can't store the object itself in the DB. What you do is to store the data from the object and reconstruct it later.

A good way is to use the excellent SQLAlchemy library. It lets you map your defined class to a table in the database. Every mapped attribute will be stored, and can be used to reconstruct the object. Querying the database returns instances of your class.

With it you can use not only sqlite, but most databases - It currently also supports Postgres, MySQL, Oracle, MS-SQL, Firebird, MaxDB, MS Access, Sybase, Informix and IBM DB2. And you can have your user choose which one she wants to use, because you can basically switch between those databases without changing the code at all.

There are also a lot of cool features - like automatic JOINs, polymorphing...

A quick, simple example you can run:

from sqlalchemy import Column, Integer, Unicode, UnicodeText, String
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base

from random import choice
from string import letters

engine = create_engine('sqlite:////tmp/teste.db', echo=True)
Base = declarative_base(bind=engine)

class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True)
    name = Column(Unicode(40))
    address = Column(UnicodeText, nullable=True)
    password = Column(String(20))

    def __init__(self, name, address=None, password=None):
        self.name = name
        self.address = address
        if password is None:
            password = ''.join(choice(letters) for n in xrange(10))
        self.password = password

Base.metadata.create_all()

Session = sessionmaker(bind=engine)
s = Session()

Then I can use it like this:

# create instances of my user object
u = User('nosklo')
u.address = '66 Some Street #500'

u2 = User('lakshmipathi')
u2.password = 'ihtapimhskal'

# testing
s.add_all([u, u2])
s.commit()

That would run INSERT statements against the database.

# When you query the data back it returns instances of your class:

for user in s.query(User):
    print type(user), user.name, user.password

That query would run SELECT users.id AS users_id, users.name AS users_name, users.address AS users_address, users.password AS users_password.

The printed result would be:

<class '__main__.User'> nosklo aBPDXlTPJs
<class '__main__.User'> lakshmipathi ihtapimhskal

So you're effectively storing your object into the database, the best way.

Answer 4

There is relatively simple way to store and compare objects, eaven to index those objects right way and to restrict (with ubique) columns containing objects. And all of that without using ORM engines. Objects mast be stored using pickle dump (so performance might be a issue) Here is example for storing python tuples, indexing restricting and comparing. This method can be easily applied to any other python class. All that is needed is explained in python sqlite3 documentation (somebody already posted the link). Anyway here it is all put together in the following example:

import sqlite3
import pickle

def adapt_tuple(tuple):
    return pickle.dumps(tuple)    

sqlite3.register_adapter(tuple, adapt_tuple)    #cannot use pickle.dumps directly because of inadequate argument signature 
sqlite3.register_converter("tuple", pickle.loads)

def collate_tuple(string1, string2):
    return cmp(pickle.loads(string1), pickle.loads(string2))

# 1) Using declared types
con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)

con.create_collation("cmptuple", collate_tuple)

cur = con.cursor()
cur.execute("create table test(p tuple unique collate cmptuple) ")
cur.execute("create index tuple_collated_index on test(p collate cmptuple)")


######################### Test ########################

cur.execute("select name, type  from sqlite_master") # where type = 'table'")
print(cur.fetchall())

p = (1,2,3)
p1 = (1,2)

cur.execute("insert into test(p) values (?)", (p,))
cur.execute("insert into test(p) values (?)", (p1,))
cur.execute("insert into test(p) values (?)", ((10, 1),))
cur.execute("insert into test(p) values (?)", (tuple((9, 33)) ,))
cur.execute("insert into test(p) values (?)", (((9, 5), 33) ,))

try:
    cur.execute("insert into test(p) values (?)", (tuple((9, 33)) ,))
except Exception as e:
    print e

cur.execute("select p from test order by p")
print "\nwith declared types and default collate on column:"
for raw in cur:
    print raw

cur.execute("select p from test order by p collate cmptuple")
print "\nwith declared types collate:"
for raw in cur:
    print raw

con.create_function('pycmp', 2, cmp)

print "\nselect grater than using cmp function:"
cur.execute("select p from test where pycmp(p,?) >= 0", ((10, ),) )
for raw in cur:
    print raw

cur.execute("select p from test where pycmp(p,?) >= 0", ((3,)))
for raw in cur:
    print raw 

print "\nselect grater than using collate:"
cur.execute("select p from test where p > ?", ((10,),) )
for raw in cur:
    print raw  

cur.execute("explain query plan select p from test where p > ?", ((3,)))
for raw in cur:
    print raw

cur.close()
con.close()

Answer 5

You can use pickle to serialize the object. The serialized object can be inserted to the sqlite DB as a bytearray field.

f=open('object.dump', 'rw')
pickle.dump(obj, f)

Now read object.dump from the file, and write it to the sqlite DB. You might want to write it as a binary data type; read about storing binary data and blob in SQLite here. Note that according to this source, SQLite limits the size of such datafield to 1Mb.

I think that a better option would be serializing your object into a file, and keeping the file name, not contents, in the database.

Answer 6

Depending on your exact needs, it could be worth looking into Django (www.djangoproject.com) for this task. Django is actually a web framework, but one of the tasks it handles is to allow you to define Models as python objects (inheriting from a base class provided by the framework). It will then automatically create the database tables required to store those objects, and sqlite is among the supported backends. It also provides handy functions to query the database and return one or more matching objects. See for example the documentation about Models in django:

http://docs.djangoproject.com/en/1.9/topics/db/models/

The drawback is of course that you have to install a full web framework, and (as far as I remember) you can only store objects whose attributes are supported by django. Also, it's made for storing many instances of predefined objects, not for storing one instance each of many different objects. Depending on your needs, this may or may not be impractical.