mmap() for large file I/O?

Question

I\'m creating a utility in C++ to be run on Linux which can convert videos to a proprietary format. The video frames are very large (up to 16 megapixels), and we need to be

Answer 1

On a 32-bit machine your process is limited to 2-3 GB of user address space. This means that (allowing for other memory use) you won't be able to map more than ~1 GB of your file at a time. This does NOT mean that you cannot use mmap() for very large files - just that you need to map only part of the file at a time.

That being said, mmap() can still be a large win for large files. The most significant advantage is that you don't waste memory for keeping the data TWICE - one copy in the system cache, one copy in a private buffer of your application - and CPU time to make those copies. It can be an even more major speedup for random access - but the "random" part must be limited in range to your current mapping(s).