问题
I've a small and simple storage system which is accessible through memory mapped files. As I need to address more than 2GB space I need a list of MappedByteBuffer with a fixed size like 2GB (I'm using less for different reasons). Then all is relativly simple: a buffer maps to a certain space say to 1GB and when I need more I map a new MappedByteBuffer (the file automatically increases), then when I need more a third buffer is mapped etc. This just worked.
But then I've read in the Java NIO book that when I change the file length problems could occur:
A MappedByteBuffer directly reflects the disc file with which it is associated. If the file is structurally modified while the mapping is in effect, strange behaviour can result (exact behaviour are OS and file system dependent) A MappedByteBuffer has a fixed size, but file it's mapped to is elastic. Specifically if a file's size changes while the mapping is in effect, some or all of the buffer may become inaccessible, undefined data could be returned, or unchecked exceptions could be thrown. Be careful about how files are manipulated by other threads or external processes when they are memory-mapped.
I think the problems could occur as the OS could move the file when it is increasing and the MappedByteBuffers then could point to an invalid space (or am I misinterpreting this?)
So, instead of adding a new MappedByteBuffer to the list I am now doing the following
- increasing the file length
- clearing the buffer list (throwing away the old buffers and hoping that the buffers where released via garbage collector. hmmh, probably I should clean all the buffers explicitly via cleaner.clean()?)
- re-mapping (filling the list with fresh buffers)
BUT THIS procedure has the disadvantage of failing sometimes while mapping with
IOException: Operation not permitted
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:734)
Why that? Because clearing the buffer list did not properly release and clean the buffers and multiple mappings are not allowed? Should I just stick to the old working method and ignoring the comment in the book?
Update
- splitting the mapping on a 32bit OS has the advantage of better finding free space and less likely make an error (ref)
- splitting the mapping into smaller parts is an advantage as setting up the mmap could be costly (ref)
- both approaches are not clean while my second approach should work but will need an unmap (will try to force the release with the normal cleaner.clean hack). The first approach should work on systems (like ibm) where I could increase the file size but in general it won't work although I couldn't find the exact reason yet...
- cleanest way would be to use multiple files I fear (one file per MappedByteBuffer)
回答1:
The root cause was my fault: accidentally I remapped the underlying file too often (the capacity increased only via mini steps).
But even in this extreme situation I was able to finally fix the IOException (Operation not permitted) when I was retrying the failing mapping operation (+ System.gc + a 5ms sleep -> which should give the jvm the opportunity to unmap the buffer). Now I only saw massive numbers of remappings which leads to the final conclusion.
At least I learned a bit more about mmap: it is very OS+filesystem dependent - thanks also to auselen! If you like a clean solution you should use one MappedByteBuffer per file as initially suggested from him. But this could be also problematic if you need large space and your OS file descriptor limit is too low.
Last but not least I would strongly recommend against my first solution as I couldn't find a guarantee (only in the IBM OS ;)) which leaves the mapped buffer intact after a file size increase.
来源：https://stackoverflow.com/questions/14011919/when-changing-the-file-length-do-i-need-to-remap-all-associated-mappedbytebuffer