How to reduce SQLite memory consumption?

Question

I\'m looking for ways to reduce memory consumption by SQLite3 in my application.

At each execution it creates a table with the following schema:

(main TE         


        

Answer 1

Assuming that all the operations in one transaction are distributed all over the table so that all pages of the table need to be accessed, the size of the working set is:

• about 1 GB for the table's data, plus
• about 1 GB for the index on the main column, plus
• about 1 GB for the original data of all the table's pages changed in the transaction (probably all of them).

---

You could try to reduce the amount of data that gets changed for each operation by moving the count column into a separate table:

CREATE TABLE main_lookup(main TEXT NOT NULL UNIQUE, rowid INTEGER PRIMARY KEY);
CREATE TABLE counters(rowid INTEGER PRIMARY KEY, count INTEGER DEFAULT 0);

Then, for each operation:

SELECT rowid FROM main_lookup WHERE main = @SEQ;
if not exists:
    INSERT INTO main_lookup(main) VALUES(@SEQ);
    --read the inserted rowid
    INSERT INTO counters VALUES(@rowid, 0);
UPDATE counters SET count=count+1 WHERE rowid = @rowid;

In C, the inserted rowid is read with sqlite3_last_insert_rowid.

Doing a separate SELECT and INSERT is not any slower than INSERT OR IGNORE; SQLite does the same work in either case.

This optimization is useful only if most operations update a counter that already exists.