We have a Java webapp that we upgraded from Java 1.5.0.19 to Java 1.6.0.21
/usr/java/jdk1.6.0_21/bin/java -server -Xms2000m -Xmx3000m -XX:MaxPermSize=256m -Djav
http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html
Note that the JVM uses more memory than just the heap. For example Java methods, thread stacks and native handles are allocated in memory separate from the heap, as well as JVM internal data structures.
So if you have a lot of threads and a lot of native handles, the memory can exceed the heap limit. Are you sure this didn't happen before as well?
Also check out this: Java using more memory than the allocated memory