How to handle an array with size 1,000,000,000 in C++?

Question

I need to handle 3D cube data. Its number of elements can be several billions. I understand I can\'t allocate that much memory on Windows. So I am thinking disk-based operat

Answer 1

Can you perhaps store the data more efficiently (read "Programming Pearls" by Bentley), is it sparse data?!

If not, memory mapped files (MMF) are your friend and allow you to map chunks of MMF into memory that you can access like any other memory.

Use CreateFileMapping and MapViewOfFile to map a chunk into your process.

Answer 2

The first and most basic step is to break the data down into chunks. The size of the chunk depends on your needs: it could be the smallest or largest chunk that can be drawn at once, or for which geometry can be built, or an optimal size for compression.

Once you're working with manageable chunks, the immediate memory problem is averted. Stream the chunks (load and unload/save) as needed.

During the load/save process, you may want to involve compression and/or a database of sorts. Even something simple like RLE and SQLite (single table with coordinates and data blob) can save a good bit of space. Better compression will allow you to work with larger chunk sizes.

Depending on usage, it may be possible to keep chunks compressed in-memory and only uncompress briefly on modification (or when they could be modified). If your data is read-only, loading them and uncompressing only when needed will be very helpful.

Splitting the data into chunks also has side-benefits, such as being an extremely simple form for octrees, allowing geometry generation (marching cubes and such) to run on isolated chunks of data (simplifies threading), and making the save/load process significantly simpler.

Answer 3

try VirtualAlloc from <windows.h>.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366887%28v=vs.85%29.aspx

its quite useful for large arrays as long as they fit in your RAM, after that 0xC0000022L's answer is probably the better solution