Using ServletOutputStream to write very large files in a Java servlet without memory issues

Question

I am using IBM Websphere Application Server v6 and Java 1.4 and am trying to write large CSV files to the ServletOutputStream for a user to download. Files are

Answer 1

I'm also not sure if flush() on ServletOutputStream works in this case, but ServletResponse.flushBuffer() should send the response to the client (at least per 2.3 servlet spec).

ServletResponse.setBufferSize() sounds promising, too.

Answer 2

Ibm websphere application server uses asynchronous data transfer for servlets by default. That means that it buffers response. If you have problems with large data and OutOfMemory exceptions, try changing settings on WAS to use synchronous mode.

Setting the WebSphere Application Server WebContainer to synchronous mode

You must also take care of loading chunks and flush them. Sample for loading from large file.

ServletOutputStream os = response.getOutputStream();
FileInputStream fis = new FileInputStream(file);
            try {
                int buffSize = 1024;
                byte[] buffer = new byte[buffSize];
                int len;
                while ((len = fis.read(buffer)) != -1) {
                    os.write(buffer, 0, len);
                    os.flush();
                    response.flushBuffer();
                }
            } finally {
                os.close();
            }

Answer 3

unrelated to your memory problems, the while loop should be:

while(bytesRead > 0);

Answer 4

your code has an infinite loop.

do
{
    bytesRead = inputStream.read(buffer, offset, buffer.length);
    resp.getOutputStream().write(buffer, 0, bytesRead);
}
while (bytesRead == buffer.length);

offset has the same value thoughout the loop, so if initially offset = 0, it will remain so in every iteration which will cause infinite-loop and which will leads to OOM error.

Answer 5

1. Kevin's class should close the m_out field if it's not null in the close() operator, we don't want to leak things, do we?

2. As well as the ServletOutputStream.flush() operator, the HttpServletResponse.flushBuffer() operation may also flush the buffers. However, it appears to be an implementation specific detail as to whether or not these operations have any effect, or whether http content length support is interfering. Remember, specifying content-length is an option on HTTP 1.0, so things should just stream out if you flush things. But I don't see that

Answer 6

Does flush work on the output stream.

Really I wanted to comment that you should use the three-arg form of write as the buffer is not necessarily fully read (particularly at the end of the file(!)). Also a try/finally would be in order unless you want you server to die unexpectedly.