Maximum amount of memory per Java process on Windows?

Question

What is the maximum heap size that you can allocate on 32-bit Windows for a Java process using -Xmx?

I\'m asking because I want to use the ETOPO1 data i

Answer 1

As noted in the question mentioned in the comment, there is a practical limit, circa 1200 MB.

However the situation you're describing has more depth to it than sheer memory size.

When you read a 910 MB binary data and build a network objects off of it (as opposed to just maintaining the data as an array of bytes), you end up consuming much more memory than 910 MB. A reasonable estimate would be that the in-memory representation will consume twice as much memory - that's because (1) each object contains an additional pointer (to the class of the object); and (2) there's a lot bookkeeping data. For instance if you use a HashMap to manage your objects then in addition to each object you also allocate a Map.Entry object which can easily consume 16 or 20 bytes (implementation dependent).

On the other hand, there's still hope: do you really need to maintain all 910 MB in memory? Can't you just build something that reads the data in a lazy manner? Combined with WeakReferences I think you can pull this off.

Answer 2

We have recently ported from Windows to Linux (because of VM size issues).

I have heard of lots of numbers thrown around in the past for Windows VM size (1200, 1400, 1600, 1800). On our Windows Servers (2003), in our environment, with our applications, ... I have never successfully used more than 1280MB. Beyond that our application started exhibiting GC and OOM issues.

Everytime I got a new VM version I tried changing the number and it never varied.

You have a 900MB file now, what if the file increases to 1300MB? What will you do?

You have a number of options

1. Port to Linux/Solaris. This just needs hardware/software and what is often a simple porting exercise.
2. Use 64bit Windows. This may not be free of GC issues though - I have heard of different tales with 64bit vms.
3. Redesign the app to process the file differently, Can you split the file logically in some way, can you read the file in chunks and process it differently etc?

Other people using OpenMap must have encountered this issue. Can you tap into their knowledge and not re-invent any wheels?

Answer 3

For a large file I suggest you use a memory mapped file. This doesn't use heap space (or very little) so maximum heap size shouldn't be a problem in this case.

Answer 4

On 32-bit Windows, by default, every application can use up to 2 GB virtual address space. I guess this makes -Xmx2048M. However, if you have more RAM installed, you can increase the virtual address space up to 3 GB by using boot time parameters.

In boot.ini, you can create a new boot options like this:

[boot loader]<br>
timeout=5<br>
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS<br>
[operating systems]<br>
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - magyar" /noexecute=optin /fastdetect /usepmtimer<br>
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - magyar 3GB" /noexecute=optin /fastdetect /usepmtimer /3GB /USERVA=2800<br>

Here by adjusting the /USERVA=2800 parameter, you can tune your machine. But be aware that some configurations don't like high values in this parameter - expect crashes.

Answer 5

There's nothing better than an empirical experiment to answer your question. I've wrote a Java program and run it while specifying the XMX flag (also used XMS=XMX to force the JVM pre-allocate all of the memory). To further protect against JVM optimizations, I've actively allocate X number of 10MB objects. I run a number of test on a number of JVMs increasing the XMX value together with increasing the number of MB allocated, on a different 32bit operating systems using both Sun and IBM JVMs, here's a summary of the results:

OS:Windows XP SP2, JVM: Sun 1.6.0_02, Max heap size: 1470 MB
OS: Windows XP SP2, JVM: IBM 1.5, Max heap size: 1810 MB
OS: Windows Server 2003 SE, JVM: IBM 1.5, Max heap size: 1850 MB
OS: Linux 2.6, JVM: IBM 1.5, Max heap size: 2750 MB

Here's the detailed run attempts together with the allocation class helper source code:

WinXP SP2, SUN JVM:

C:>java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b06)
Java HotSpot(TM) Client VM (build 1.6.0_02-b06, mixed mode)


java -Xms1470m -Xmx1470m Class1 142
...
about to create object 141
object 141 created

C:>java -Xms1480m -Xmx1480m Class1 145
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

WinXP SP2, IBM JVM

 
C:>c:\ibm\jdk\bin\java.exe -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pwi32devifx-20070323 (if
ix 117674: SR4 + 116644 + 114941 + 116110 + 114881))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223ifx-2007
0323 (JIT enabled)
J9VM - 20070322_12058_lHdSMR
JIT  - 20070109_1805ifx3_r8
GC   - WASIFIX_2007)
JCL  - 20070131


c:\ibm\jdk\bin\java.exe -Xms1810m -Xmx1810m Class1 178
...
about to create object 177
object 177 created

C:>c:\ibm\jdk\bin\java.exe -Xms1820m -Xmx1820m Class1 179
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate he
ap.  1820M requested
Could not create the Java virtual machine.

Win2003 SE, IBM JVM

C:>"C:\IBM\java" -Xms1850m -Xmx1850m Class1
sleeping for 5 seconds.
Done.


C:>"C:\IBM\java" -Xms1880m -Xmx1880m
 Class1
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate he
ap.  1880M requested
Could not create the Java virtual machine.

Linux 2.6, IBM JVM

[root@myMachine ~]# /opt/ibm/java2-i386-50/bin/java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pxi32dev-20060511 (SR2))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20060504 (JIT enabled)
J9VM - 20060501_06428_lHdSMR
JIT  - 20060428_1800_r8
GC   - 20060501_AA)
JCL  - 20060511a


/opt/ibm/java2-i386-50/bin/java -Xms2750m -Xmx2750m Class1 270

[root@myMachine ~]# /opt/ibm/java2-i386-50/bin/java -Xms2800m -Xmx2800m Class1 270
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap.  2800M requested
Could not create the Java virtual machine.

Here's the code:

import java.util.StringTokenizer;


public class Class1 {

        public Class1() {}

        private class BigObject {
                byte _myArr[];
                public BigObject() {
                        _myArr = new byte[10000000];
                }
        }
    public static void main(String[] args) {
                (new Class1()).perform(Integer.parseInt(args[0]));
        }
        public void perform(int numOfObjects) {
                System.out.println("creating 10 MB arrays.");
                BigObject arr[]  = new BigObject[numOfObjects];
                for (int i=0;i <numOfObjects; i++) {
                        System.out.println("about to create object "+i);
                        arr[i] = new BigObject();
                        System.out.println("object "+i+" created");
                }
                System.out.println("sleeping for 5 seconds.");
                try {
                Thread.sleep(5000);
                }catch (Exception e) {e.printStackTrace();}
                System.out.println("Done.");
    }

}