Does mmap really copy data to the memory?

Question

It is said that mmap() maps files to the memory, and it costs to the virtual address space memory of the calling process. Does it really copy data to the memory, or

Answer 1

The data still exists on the disk. The OS allocates some physical memory and copies file data into it so you access the file contents there (that happens when you try to access file data). That physical memory is mapped into the virtual address space of the process. The OS can unmap long unused portions of the file and it will map them back when needed. If there's little free physical memory, unmaps can be more aggressive, leading to poor performance.

Memory mapping of files:

1. uses less physical memory and virtual address space than simple "file reads/writes" because there are no file buffers here and there (in the OS, in the C standard library and in your program) and there's no unnecessary copying between them.

2. can be (and probably is, if you have enough free physical memory in certain conditions, depending on how much data we're talking about and how much physical memory the OS lets us use for mmap'ing) faster than simple "file reads/writes" because of the above and because you avoid transitions between the user and kernel modes that the "file read" system call involves. The only transitions that remain are those to map a specific page that's currently unmapped, they are initiated by page faults (=CPU exceptions), which are handled in the kernel. As long as everything you need is mapped, there are no user-kernel transitions while accessing file data.

Answer 2

Copying does not imply that the original is destroyed.

It maps the contents of the disk into memory, so of course at some point the bits must be copied, yes.

And since this means it needs address space, that occupies part of the process' virtual address space.

Answer 3

The only thing the mmap function really does is change some kernel data structures, and possibly the page table. It doesn't actually put anything into physical memory at all. After you call mmap, the allocated region probably doesn't even point to physical memory: accessing it will cause a page fault. This kind of page fault is transparently handled by the kernel, in fact, this is one of the kernel's primary duties.

What happens with mmap is that the data remains on disk, and it is copied from disk to memory as your process reads it. It can also be copied to physical memory speculatively. When your process gets swapped out, the pages in the mmap region do not have to be written to swap because they are already backed by long-term storage -- unless you have modified them, of course.

However, mmap will consume virtual address space, just like malloc and other similar functions (which mostly use mmap behind the scenes, or sbrk, which is basically a special version of mmap). The main difference between using mmap to read a file and read to read a file is that unmodified pages in an mmap region do not contribute to overall memory pressure, they are almost "free", memory wise, as long as they are not being used. In contrast, files read with the read function will always contribute to memory pressure whether they are being used or not, and whether they have been modified or not.

Finally, mmap is faster than read only in the use cases which it favors -- random access and page reuse. For linearly traversing a file, especially a small file, read will generally be faster since it does not require modifying the page tables, and it takes fewer system calls.

As a recommendation, I can say that any large file which you will be scanning through should generally be read in its entirety with mmap on 64-bit systems, and you can mmap it in chunks on 32-bit systems where virtual memory is less available.

See also: mmap() vs. reading blocks

See also (thanks to James): When should I use mmap for file access?

Answer 4

"virtual memory" of a process is the range of addresses available to it. To make something available in memory, you need to reserve a range of addresses, so mmap() takes up some virtual memory.

Under Linux (and many other systems probably use similar mechanism), when reading a file, the content is first read into memory allocated by kernel (in Linux this is the "page cache"). Than if you use mmap(), this memory is simply made available to the process by assigning it some address in that process' address space. If you use read(), the process allocates a buffer, which needs both addresses (virtual memory) and a place to live (physical memory) and the data get copied from the page cache to that buffer (more physical memory is needed).

The data is only read from disk when actually accessed. In mmap() case it means when you actually address the memory, in read() it is the copy to your buffer, so inside the read() call.

Thus mmap() is more efficient for large files, especially for random access. The disadvantages are that it can only be used for files and not file-like objects (pipes, sockets, devices, /proc files etc.) and that an IO failure is detected during the page-fault, where they are difficult to handle (it sends SIGBUS signal), while read can return error and the application can try to recover (most don't anyway). The later is mainly concern for network filesystems where IO failure might be because of lost connection.