How to persist data between executions in Python

Question

I am working on a personal project in Python where I need some form of persistent data. The data would fit in 2-3 tables of 10-20 columns and 100-200 records each. I have

Answer 1

Or, if you just want to persist data between executions - for such a small data set you could have a look at the pickle module for persistency, and just load the data into memory during execution.

It's a simple solution - but for a personal project it might be enough.

Answer 2

I agree with using sqlite3. It is very easy to use, you don't need to worry about having to set up a database server. You should check out the SQLAlchemy library too.

Answer 3

This sounds like very few data. An SQL DB might be overkill, especially with an ORM on top. I'd check whether JSON could do the job...

Answer 4

You should use sqlite3 module for this, it is included in Python.

Also you may want too look for an ORM solution.

Answer 5

Peewee is another ORM that works with SQLite. It is an alternative to SQLAlchemy. If using SQLite, I would consider Peewee for pet projects and SQLAlchemy for professional work. I typically would not use SQLite directly.

Answer 6

The real question is really what kind of operations you want to do with your data.

As far as storage possibilities, the simplest solutions are indeed sqlite3 and pickle.

The solution that you will choose depends basically on whether using SQL or Python is the easiest way for you to manage your data. SQL is probably better at complex operations than Python, but Python is definitely more lightweight and simpler, and therefore is a good choice for simple operations. So, if using pickle+Python is too cumbersome, then sqlite3 is a very good choice.