How to compress a .net object instance using gzip

Question

I am wanting to compress results from QUERYS of the database before adding them to the cache.

I want to be able to compress any reference type.

I have a working

Answer 1

Here is how this can be done.

GzipStream: Provides methods and properties used to compress and decompress streams.

Serialize object to Memory stream using binaryformatter and hook that memorystream to Gzipstream.

Code to compress is as follows:

            public static byte[] ObjectToCompressedByteArray(object obj)
                    {
                        try
                        {
                            using (var memoryStream = new System.IO.MemoryStream())
                            {
                                using (var gZipStream = new GZipStream(memoryStream, CompressionMode.Compress))
                                {
                                    var binaryFormatter = new BinaryFormatter();
                                    binaryFormatter.Serialize(gZipStream, obj);
                                }
                                return memoryStream.ToArray();
                            }
                        }
                        catch (Exception ex)
                        {
                            LoggerWrapper.CMLogger.LogMessage(
                                $"EXCEPTION: BSExportImportHelper.ObjectToByteArray - : {ex.Message}", LoggerWrapper.CMLogger.CMLogLevel.Error);
                            throw;
                        }

                    }

Answer 2

What sort of objects are you putting in the cache? Are they typed objects? Or things like DataTable? For DataTable, then perhaps store as xml compressed through GZipStream. For typed (entity) objects, you'll probably need to serialize them.

You could use BinaryFormatter and GZipStream, or you could just use something like protobuf-net serialization (free) which is already very compact (adding GZipStream typically makes the data larger - which is typical of dense binary). In particular, the advantage of things like protobuf-net is that you get the reduced size without having to pay the CPU cost of unzipping it during deserialization. In some tests before adding GZipStream, it was 4 times faster than BinaryFormatter. Add the extra time onto BinaryFormatter for GZip and it should win by a considerable margin.

Answer 3

I just added GZipStream support for my app today, so I can share some code here;

Serialization:

using (Stream s = File.Create(PathName))
{
    RijndaelManaged rm = new RijndaelManaged();
    rm.Key = CryptoKey;
    rm.IV = CryptoIV;
    using (CryptoStream cs = new CryptoStream(s, rm.CreateEncryptor(), CryptoStreamMode.Write))
    {
        using (GZipStream gs = new GZipStream(cs, CompressionMode.Compress))
        {
            BinaryFormatter bf = new BinaryFormatter();
            bf.Serialize(gs, _instance);
        }
    }
}

Deserialization:

using (Stream s = File.OpenRead(PathName))
{
    RijndaelManaged rm = new RijndaelManaged();
    rm.Key = CryptoKey;
    rm.IV = CryptoIV;
    using (CryptoStream cs = new CryptoStream(s, rm.CreateDecryptor(), CryptoStreamMode.Read))
    {
        using (GZipStream gs = new GZipStream(cs, CompressionMode.Decompress))
        {
            BinaryFormatter bf = new BinaryFormatter();
            _instance = (Storage)bf.Deserialize(gs);
        }
    }
}

NOTE: if you use CryptoStream, it is kinda important that you chain (un)zipping and (de)crypting right this way, because you'll want to lose your entropy BEFORE encryption creates noise from your data.

Answer 4

This won't work for any reference type. This will work for Serializable types. Hook up a BinaryFormatter to a compression stream which is piped to a file:

var formatter = new BinaryFormatter();
using (var outputFile = new FileStream("OutputFile", FileMode.CreateNew))
using (var compressionStream = new GZipStream(
                         outputFile, CompressionMode.Compress)) {
   formatter.Serialize(compressionStream, objToSerialize);
   compressionStream.Flush();
}

You could use a MemoryStream to hold the contents in memory, rather than writing to a file. I doubt this is really an effective solution for a cache, however.