mmap() for large file I/O?

Question

I\'m creating a utility in C++ to be run on Linux which can convert videos to a proprietary format. The video frames are very large (up to 16 megapixels), and we need to be

Answer 1

On a 32-bit machine your process is limited to 2-3 GB of user address space. This means that (allowing for other memory use) you won't be able to map more than ~1 GB of your file at a time. This does NOT mean that you cannot use mmap() for very large files - just that you need to map only part of the file at a time.

That being said, mmap() can still be a large win for large files. The most significant advantage is that you don't waste memory for keeping the data TWICE - one copy in the system cache, one copy in a private buffer of your application - and CPU time to make those copies. It can be an even more major speedup for random access - but the "random" part must be limited in range to your current mapping(s).

Answer 2

If your files are 10 GB or more, then don't even think about trying to use mmap() on a 32-bit architecture. Go directly to a 64-bit OS, which should be able to handle it just fine.

Note that files that are mapped into memory space don't actually consume the same amount of RAM (as the file size), so you won't need to install hundreds of gigabytes of RAM in your machine.