Using ServletOutputStream to write very large files in a Java servlet without memory issues

Question

I am using IBM Websphere Application Server v6 and Java 1.4 and am trying to write large CSV files to the ServletOutputStream for a user to download. Files are

Answer 1

The average decent servletcontainer itself flushes the stream by default every ~2KB. You should really not have the need to explicitly call flush() on the OutputStream of the HttpServletResponse at intervals when sequentially streaming data from the one and same source. In for example Tomcat (and Websphere!) this is configureable as bufferSize attribute of the HTTP connector.

The average decent servletcontainer also just streams the data in chunks if the content length is unknown beforehand (as per the Servlet API specification!) and if the client supports HTTP 1.1.

The problem symptoms at least indicate that the servletcontainer is buffering the entire stream in memory before flushing. This can mean that the content length header is not set and/or the servletcontainer does not support chunked encoding and/or the client side does not support chunked encoding (i.e. it is using HTTP 1.0).

To fix the one or other, just set the content length beforehand:

response.setContentLengthLong(new File(path).length());

Or when you're not on Servlet 3.1 yet:

response.setHeader("Content-Length", String.valueOf(new File(path).length()));

Answer 2

I have used a class that wraps the outputstream to make it reusable in other contexts. It has worked well for me in getting data to the browser faster, but I haven't looked at the memory implications. (please pardon my antiquated m_ variable naming)

import java.io.IOException;
import java.io.OutputStream;

public class AutoFlushOutputStream extends OutputStream {

    protected long m_count = 0;
    protected long m_limit = 4096; 
    protected OutputStream m_out;

    public AutoFlushOutputStream(OutputStream out) {
        m_out = out;
    }

    public AutoFlushOutputStream(OutputStream out, long limit) {
        m_out = out;
        m_limit = limit;
    }

    public void write(int b) throws IOException {

        if (m_out != null) {
            m_out.write(b);
            m_count++;
            if (m_limit > 0 && m_count >= m_limit) {
                m_out.flush();
                m_count = 0;
            }
        }
    }
}

Answer 3

So, following your scenario, shouldn't you been flush(ing) inside that while loop (on every iteration), instead of outside of it? I would try that, with a bit larger buffer though.

Answer 4

The while condition does not work, you need to check the -1 before using it. And please use a temporary variable for the output stream, its nicer to read and it safes calling the getOutputStream() repeadably.

OutputStream outStream = resp.getOutputStream();
while(true) {
    int bytesRead = inputStream.read(buffer);
    if (bytesRead < 0)
      break;
    outStream.write(buffer, 0, bytesRead);
}
inputStream.close();
out.close();