How to reduce the size of an sqlite3 database for iphone?

Question

edit: many thanks for all the answers. Here are the results after applying the optimisations so far:

• Switching to sorting the characters and run l

Answer 1

Remove the indexes on sowpods and twl -- they are probably not helping your query times and are definitely taking lots of space.

You can get stats on the database using sqlite3_analyzer from the SQLite downloads page.

Answer 2

Do I reckon correctly that you have about 450K words like that in your database ?

I've got no clue about iPhone, neither serious about sqlitem but... as long as sqlite does not allow for a way to save the file as gz right away (it maybe already does internally? no, does not look like that when you say it's about 135 b per entry. not even with both indexes), I would move away from the table approach, save it "manually" in a dictionary approach compression and build the rest on the fly and in memory. That should perform VERY well on your type of data.

Wait... Are you using that signature to allow for fulltextsearching or mistyping recogition ? Would full text search on sqlite not obsolete that field ?

Answer 3

As noted storing "Signature" more efficiently seems like a good idea.

However, it also seems like you could gain a ton of space savings by using some kind of lookup table for words - since you seem to be taking a root word and then appending "er", "ed", "es", etc why not have a column with a numeric ID that references a root word from a separate lookup table, and then a separate column with a numeric ID that references a table of common word suffixes that would be appended to the base word.

If there were any tricks around storing shorthand versions of signatures for multiple entries with a single root word, you could also employ those to reduce the size of stored signatures (not sure what algorithm is producing those values)

This also seems to make a lot of sense to me as you have the "word" column as a primary key, but do not even index it - just create a separate numeric column that is the primary ID for the table.

Answer 4

Your best bet is to use compression, which unfortunately SQLite does not support natively at this point. Luckily, someone took the time to develop a compression extension for it which could be what you need.

Otherwise I'd recommend storing your data mostly in compressed format and uncompressing on the fly.

Answer 5

I'm not clear on all the use cases for the signature field but it seems like storing an alphabetized version of the word instead would be beneficial.

Answer 6

Have you tried typing the "vacuum" command to make sure you don't have extra space in the db you forgot to reclame?