There are -Xmx and max-heap-size (jnlp) options to set maximum memory java application can use.
If these options are not specified application is permitted to use only s
-XX:DefaultMaxRAMFraction=1
bugs.sun.com
Sun's 1.5 JVM
Default Heap Size
Doesn't work for web start.
Unfortunately, the best way I'm aware of is to write a simple launcher (possibly a shell script or batch file) that inspects the system, determines an appropriate value for -Xmx, and then launches the Java application. Once the JVM has initialized, it is not possible to change the values.