Sometimes 32-bit applications run faster, sometimes they run slower. There are a lot of competing factors affecting the performance. Generally the difference in either direction isn't very large.
I don't see how you can legitimately exclude any factor from performance considerations, because performance always happens in the real world where all factors are in play. Speed is not theoretical and cannot be treated in isolation.
Here is an article where the author did some benchmarking:
http://www.osnews.com/story/5768
He found that the 32-bit binaries were faster in his tests. That was on SPARC in early 2004.
Here is a 2010 article from Intel about moving to 64-bit applications:
http://software.intel.com/en-us/articles/moving-from-32-bit-applications-to-64-bit-applications-en/
According to this article, 64-bit binaries can giver you faster math, so math-heavy code should be faster if correctly optimized. Otherwise, your 64-bit code may well be slower, particular since it takes more memory and this may result in more time spent accessing RAM (more frequent cache misses).
