One of the biggest reasons Java and other Virtual Machine languages are not used for games is due to Garbage Collection. The same thing goes for .NET. Garbage collection has come a long ways and works great in most types of applications. In order to do garbage collection though, you do need to pause and interrupt the application to collect the trash. This can cause periodic lag when collection happens.
Java has the same problem for realtime applications. When tasks must run at a specific time, it is hard to have an automated task such as garbage collection respect that.
It is not that Java is slow. It is that Java is not good at handling realtime tasks.
