Very large array on the heap (Visual C++)

Question

I hope some one can help me, i'm trying to create an int[400000000] (400 millions) array on my application using visual c++ 2010 but it generates an overflow error The same code runs on linux with g++. I need this because i'm working with large matrices. Thank you in advance.

Answer 1

If you are using a 32-bit application then by default you have just 2GB of user address space. 400 million integers is about 1.5GB. You are very likely not to have this much contiguous address space. It is possible to force 32-bit windows to allocate a 3GB user address space for each process but this may just be a stop gap for your situation.

If you can move to a 64-bit architecture then this should not be an issue; otherwise you should find a way of storing your matrix data in a way that does not require a single block of contiguous storage, for example storing it in chunks.

Answer 2

I think what you need is a Divide-and-Conquer algorithm. Not memory space.

Answer 3

I'm not sure if in you're case it wouldn't even be better to use STXXL.

Answer 4

Perhaps sparse matrices are of use in your application. This concept is used when dealing with big matrices which have a lot of 0 entries, which can be the case in quite a lot of applications.

And by the way, you do not gain anything by storing such a huge amount of data on the heap. Consider, that your CPU cache has perhaps 12 MB! At least use some intelligent dynamic memory allocation mechanism.

Answer 5

Does the whole array really needs to be allocated ? do you really use the whole array ? Is it an array with lots of 0 ? if it is the case, then the fact that it works better on linux can be explained.

In that case using a sparse array might be more appropriate. Using an existing sparse array implementation would reduce the memory footprint and maybe allow faster computation.

Answer 6

I just found a very simple solution but i don't know if it is advisable

int tab[400000000]={0};//global array

int main(array<System::String ^> ^args)
{
std::cout<<tab[399999999]<<std::endl;//ok

/*
int* tab=new int[400000000];//doesn't work
...
delete[] tab;
*/
    return 0;
}