Yes. Java3D, LWJGL and JOGL support GLSL (OpenGL Shading Language).
Edit:
You can use OpenCL if you want to do platform-neutral, general-purpose computation on GPUs. This framework lets you write code that treats all processing units identically, despite wildly varying feature sets and execution environments. Though, this is very low level programming compared with Java.
It seems your ideal would be a JVM written with OpenCL support. Searching online, I found a little bit of interest in the idea but no evidence of any major backing.
how much of a performance improvement can one expect
That depends on the system you're running on and what sort of data you're processing (Matrix and vector math is extremely efficient on GPUs). You'd likely get some major gains on a system like mine with two powerful GPUs and a modest single-core CPU. However on a computer with a modest GPU and a quad-core CPU, the performance gains might have a hard time overcoming the overhead.
