How to make Apache mod_deflate and Transfer-encoding : Chunked work together?

Question

I am trying to use the bigpipe concept on our website. That means trying to send the response in chunks instead of sending it as a whole so that user feels that page is fast. I am successful in doing that by using the flushBuffer method on the response object in java. But now when I try to compress the content with apache mod_deflate module, chunking is lost.

Here is the configuration from apache used to compress the content

**

Begin mod_deflate config

DeflateBufferSize 100
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
DeflateFilterNote Input input_info
DeflateFilterNote Output output_info
DeflateFilterNote Ratio ratio_info
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/httpd/deflate_log deflate

End mod_deflate config**

Here is the response header when the deflate is turned on in apache

Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:7916
Content-Type:text/html; charset=UTF-8
Date:Fri, 27 Jan 2012 20:11:11 GMT
Keep-Alive:timeout=300, max=3997
Server:Apache
Vary:Accept-Encoding

Response header when the deflate is turned off in apache

Connection:Keep-Alive
Content-Type:text/html; charset=UTF-8
Date:Fri, 27 Jan 2012 20:21:14 GMT
Keep-Alive:timeout=300, max=3997
Server:Apache/2.2.3 (CentOS)
Transfer-Encoding:chunked

As you can see in above 2 headers chunking is working only if the compression is turned off. I was searching on internet regarding this and people were suggesting to decrease the DeflateBufferSize value. I decreased the value to 100 bytes as you can see in my apache config but that still didn't solve the problem. DeflateBufferSize set to 100 bytes means that response is buffered in apache till 100 bytes are received and then the it is compressed.

I was looking at the mod_gzip module which was bundled with the old apache 1.3 and that module has a following directive which allows chunked content to be gzipped.

mod_gzip_dechunk Yes

Does anyone know of such directive in mod_deflate bundled with apache 2.x?

Or Does anyone know how to compress the chunked content?

Answer 1

Actually I found the solution. I used to create a new object of GZipOutputStream each time to flush different chunks. Instead you should create one object only of GZipOutputStream and then used that object for compressing all the chunks of the response. Also I put a wrapper around GZipOutputStream. Here is the wrapper that I got from googling around.

public class GZIPFlushableOutputStream extends GZIPOutputStream {

    public GZIPFlushableOutputStream(final OutputStream out) throws IOException {
        // Using Deflater with nowrap == true will ommit headers and trailers
        super(out);
    }

    private static final byte[] EMPTYBYTEARRAY = new byte[0];

    /**
     * Insure all remaining data will be output.
     */
    public void flush() throws IOException {
        /**
         * Now this is tricky: We force the Deflater to flush its data by
         * switching compression level. As yet, a perplexingly simple workaround
         * for
         * 
         * http://developer.java.sun.com/developer/bugParade/bugs/42557 43.html
         */
        def.setInput(EMPTYBYTEARRAY, 0, 0);

        def.setLevel(Deflater.NO_COMPRESSION);
        deflate();

        def.setLevel(Deflater.DEFAULT_COMPRESSION);
        deflate();

        out.flush();
    }
}

Answer 2

My understanding is that you need the "whole" file in order to compress it. You can either send it out in chunks or send it compressed. The mod_gzip_dechunk option does not appear to exist any more - see mod_deflate documentation.