How to compress small strings

Question

I have an sqlite database full of huge number of URLs and it\'s taking huge amount of diskspace, and accessing it causes many disk seeks and is slow. Average URL path length is

Answer 1

Use the compress algorithm but use a shared dictionary.

I've done something like this before where I used the LZC/LZW algorithm, as used by the Unix compress command.

The trick to get good compression with short strings is to use a dictionary made up of a standard sample of the URLs you are compressing.

You should easily get 20%.

Edit: LZC is a variant of LZW. You only require LZW as you only need a static dictionary. LZC adds support for resetting the dictionary/table when it gets full.