I know about the -XX:+HeapDumpOnOutOfMemoryError JVM parameter. I also know about -XX:OnOutOfMemoryError=\"cmd args;cmd args\" and that kill -3 will
kill -3
I bet the runtime sets a specific errorlevel on crash. Check for that return code and rerun the program in that case. You should perhaps put that into a script.
The sun jre allows you to heap dump on oome, perhaps openjdk does too.