We ship a Java application whose memory demand can vary quite a lot depending on the size of the data it is processing. If you don\'t set the max VM (virtual memory) size
Have you looked at running jps to give you the PID for your process and then calling jinfo to change the mx option? Not sure if this will work but it may.
[Edit] This would mean that when you think you have a big dataset, you read the total amount of ram somehow (OS dependent I think. See http://forums.sun.com/thread.jspa?messageID=10306570) or you just increase the size until you don't think it is low anymore (if it blows up first, try to capture and display a helpful message such as "your machine is inadequate, time to make a run to Frys").
The max VM sizes indeed answer to that need (it sets the max value, but the VM will take only necessary, step by step), but if you need several configurations, besides supplying different "cmd" files, I don't really see a way (though i'll search a bit more)
[edit] How about using a first program/script (or even another java program), which would check the available resources for the system, and then only call your program with the appropriate -Xm, according to what it retrieved from system ? That way it would adapt to machines, even if you don't know them before. Could be an idea...
[second edit] Ok, this has been proposed already by skaffman, my bad.