Lots of people make absolute statements about things, because they don't actually think. They'd rather just apply a rule, making things more tedious, but requiring less design and forethought. I'd rather have a bit of hard thinking now and then when I'm doing something hairy, and abstract away the tedium, but I guess not everyone thinks that way. Sure, smart pointers have a performance cost. So do exceptions. That just means there may be some small portions of your code where you shouldn't use them. But you should profile first and make sure that's actually what the problem is.
Disclaimer: I've never done any game programming.
