Following on from a previous question relating to heap usage restrictions, I\'m looking for a good standard C++ class for dealing with big arrays of data in a way that is bo
std::deque does exactly what you're describing, but usually at the granularity of the OS page size (that is, the chunks it allocates are usually 4 kB).
If you're unhappy with deque's default performance, you might be able to write a custom allocator that grabs bigger chunks--that is, gets 1 MB or more at a time.
As others have said, your process's virtual address space is completely independent of all other processes, so you can address 2GB no matter what else is going on in your system. The OS will swap your memory pages to/from disk as necessary to fit the constraints of the amount of installed memory and all the processes contending for it. This will happen at the 4 kB page size, independent of how big your chunks are.
Exactly how sparse is this array? If there are large amounts of empty (unused) space in it, you may want to take another approach. The answer to this question suggests an stl map.
If it isn't sparse (as mentioned in the comments), one thing you might look into since you are running on Windows is using a memory-mapped file. Although your OS may be 32-bit, your file system is not. This does of course mean there will be swapping going on though, which is liable to be quite a bit slower than if you could really put the whole darn thing in RAM.
Also, you really ought to consider knocking the system's RAM up to the max (3GB on 32-bit Windows I believe) to see if that fixes it for you. That should only cost you about $100, and you are spending way more than that in man-hours just worrying about this.
Have you tried using an std::deque
? Unlike a std::vector
, which uses one huge heap allocation, deque
usually allocates in small chunks, but still provides amortised constant time indexing via operator[]
.
From the point of view of your program you always have 2GB available on startup, no matter what else is going on in the system. I don't believe Windows provides a way to detect if you have memory being paged out to disk or not. As far as you data structures go, it sounds like you're describing something similar to how a deque is implemented in the STL.