Prevent OutOfMemory when using java.nio.MappedByteBuffer

问题

Consider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size.

MappedByteBuffer b = ch.map(FileChannel.MapMode.READ_ONLY, r, 1024*1024*5);

Sooner or later, when application works with big files, oom is thrown

java.io.IOException: Map failed  at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)

According to specification, MappedBuffer should dispose direct memory as soon as it is GC itself. Looks like the problem is, that MappedBuffer-s are GC-ed too late, later then direct memory finished.

How to avoid this situation ? Probably say MappedBuffer to dispose implicitly or use some kind of pool of MappedBuffer

回答1:

You can avoid having to trigger a GC by cleaning up the mapped byte buffers directly.

public static void clean(ByteBuffer bb) {
    if(bb == null) return;
    Cleaner cleaner = ((DirectBuffer) bb).cleaner();
    if(cleaner != null) cleaner.clean();
}

Provided you call this before discarding, you won't run out of virtual memory.

Perhaps you can look at creating larger ByteBuffers less often (unless you have a large number of files) Creating a MappedByteBuffer is not free (takes about 50 micro-seconds on some machines)

回答2:

The error message says "map failed", not "heap space" or "permgen space". This means the JVM doesn't have enough address space available.

See this bug in Sun's database, and also this question.

The first link provides a workaround (ewww) which is close the what the second link says:

    try {
        buffer = channel.map(READ_ONLY, ofs, n);
    } catch (java.io.IOException e) {
        System.gc();
        System.runFinalization();
        buffer = channel.map(READ_ONLY, ofs, n);
    }

回答3:

Maybe a WeakHashMap to pool those MappedBuffers would work.

But before you guess about the root cause, I'd recommend hooking your app up to Visual VM 1.3.3, with all the plugins installed, so you can see exactly what's causing the OOM error. You're presuming that these MappedBuffers are doing it, but they're only 5MB each for 5-6 threads - 25-30MB total.

Better to have data than guess. Visual VM will get it for you.