Java Out of memory 2D Array

问题

I am trying to create a 2D array as below.

int NUM_RECORDS = 100480507;

byte[][] completeArray = new byte[NUM_RECORDS][6];

Shouldn't it be enough to have 100480507 * 6 ~= 0.6 GB

See this question as well.

But creation of this array runs out of memory. I have allocated 4G to my java process through JVM args.

How can that be explained? I am I missing some thing trivial here?

This is my program

public class MemTest {

   public static void main(String[] args) {
       int NUM_RECORDS = 100480507;
       byte[][] completeArray = new byte[NUM_RECORDS][6];
       System.out.println("Array created");
   }
}

回答1:

Each array has an overhead (for example, see IBM documentation on their overhead -> http://www.ibm.com/developerworks/java/library/j-codetoheap/index.html). In your case, you are creating 100480507 of them!

If you change your code to "byte[] completeArray = new byte[NUM_RECORDS*6];", according to your theory, it should demand the same space! But, I am fairly certain this will work as there is least overhead. You could also try "byte[][] completeArray = new byte[6][NUM_RECORDS];" and that should work too (lesser overhead).

I know this will not solve your issue - but I hope this will give you some perspective on the overhead.

回答2:

I check the object layout in Harmony JVM (Other implementations of JVM are similar I suppose). Each Object in java has a object header that contains important information for JVM. The most important is a reference to the object's class (one word). Moreover, there are some flags used by the GC and to manage synchronization, i.e., lockword (since every object can be synchronized on) which takes up another one word (using partial words would be bad for performance). So that's 2 words, which is 8 bytes on 32 bit systems, and 16 bytes on 64 bit. Arrays additionally need an int field for the array length, which is another 4 bytes, possibly 8 bytes on 64 bit systems. So for each array, we have 12 additional bytes on 32-bit machine, and may be 24 bytes on 64-bit machine.

In your program, you have 6 arrays of array. 100480507 1D array. So the additional memory consumption is about 1.2 + 0.6 GB, which is a very large block of continous memory. The overhead is about 200%.

When we change the code to:

public class MemTest {
   public static void main(String[] args) {
       int NUM_RECORDS = 100480507;
       byte[] completeArray = new byte[NUM_RECORDS * 6];
       System.out.println("Array created");
   }
}

We only create 1 array, so the overhead is very small. About 0.6GB in total.

When the code changed to :

public class MemTest {
   public static void main(String[] args) {
       int NUM_RECORDS = 100480507;
       byte[][] completeArray = new byte[NUM_RECORDS][6];
       System.out.println("Array created");
   }
}

We got 7 arrays in total. The program will end immediately.

Create another 0.6GB memory in your code style, but change the element type from int to long:

public class MemTest {
   public static void main(String[] args) {
       int NUM_RECORDS = 12560063;
       long[][] completeArray = new long[NUM_RECORDS][6];
       System.out.println("Array created");
   }
}

The program can also end immediately. The overhead is about 150M/600M = 25%.

来源：https://stackoverflow.com/questions/14641794/java-out-of-memory-2d-array

标签

java

memory-management

bytearray

out-of-memory