A large reason is that video games require direct knowledge of the hardware underneath, often times, and there really is no great implementation for many architectures. It's the knowledge of the underlying hardware architecture that allows developers to squeeze every ounce of performance out of a gaming system. Why would you take the time to port Java to a gaming platform, and then write a game on top of that port when you could just write the game?
edit: this is to say that it's more than a "speed" or "don't have the right libraries" issue. Those two things go hand-in-hand with this, but it's more a matter of "how do I make a system like the cell b.e. run my java code? there aren't really any good java compilers that can manage the pipelines and vectors like i need.."