Java multithreading reading a single large file

Question

What is an efficient way for a Java multithreaded application where many threads have to read the exact same file (> 1GB in size) and expose it as an input stream? I\'ve notice

Answer 1

A few ideas:

1. Write a custom InputStream implementation that acts as a view onto a FileChannel. Write this such that it does not rely on any state in the FileChannel. (ie: each instance should keep track of its own position and reading should use absolute reads on the underlying FileChannel.) This at least gets you around the trouble you had with Channels.newInputStream(), but it may not solve your IO contention issues.

2. Write a custom InputStream implementation that acts as a view onto a MappedByteBuffer. Memory mapping shouldn't be as bad as actually reading the whole thing into memory at once, but you'll still eat up 1GB of virtual address space.

3. Same as #1, but have some sort of shared caching layer. I wouldn't try this unless 1 turns out to not be efficient enough and 2 isn't feasible. Really, the OS should already be doing some caching for you in #1, so here you're essentially trying to be smarter than the OS filesystem caching.