Implementing in-memory compression for objects in Java

Question

We have this use case where we would like to compress and store objects (in-memory) and decompress them as and when required.

The data we want to compress is quite

Answer 1

This is a tricky problem:

First, using ObjectOutputStream is probably not the answer. The stream format includes a lot of type-related metadata. If you are serializing small objects, the mandatory metadata will make it hard for the compression algorithm to "break even", even if you implement custom serialization methods.

Using DataOutputStream with minimal (or no) added type information will give a better result, but mixed data is not generally that compressible using a general purpose compression algorithms.

For better compression, you may need to look at the properties of the data that you are compressing. For instance:

• Date objects could be represented as int values if you know that have a precision of 1 day.
• Sequences of int values could be run-length encoded, or delta-encoded if they have the right properties.
• and so on.

---

However way you do it, you will need to do a serious amount of work to get a worthwhile amount of compression. IMO, a better idea would be to write the objects to a database, datastore or file and use caching to keep frequently used objects in memory.

Answer 2

There are various compression algorithm implemented in the JDK. Check the [java.util.zip](http://download.oracle.com/javase/6/docs/api/java/util/zip/package-summary.html) for all algorithm implemented. However it may not be a good thing to compress all your data. For instance a serialized empty array may be several dozen of bytes long as the name of the underlying class is in the serialized data stream. Also most compression algorithm are designed to remove redundancy from large data blocks. On small to medium Java objects you'll probably have very little or no gain at all.