I know everyone who reads the question will think \"Firebug!\" right away. Maybe some will think \"YSlow!\" and \"Google Page Speed!\"
While I really like these tools, I
I really like AOL's PageTest tool, it combines a lot of the utility found in Firebug, YSlow, and PageTest and wraps it up in a nice web interface with a few nice features. First, it can be run on IE7 or IE8 (no 6, sorry) from the US or international locations to give you a better idea of performance there. It provides waterfall charts like those in the Firebug net panel which are useful for determining where time is spent. It also provides recommendations for what to fix which are similar to those in YSlow. Finally, it allows you to run multiple trials on the same site so that you can get more accurate results while minimizing the external factors.
For the most part I would agree with OrbMan in that it is important to optimize for all browsers. Since those optimizations are cross-browser you will get the most bang for you buck in terms of performance. Only then is it time to start considering browser-specific improvements.
There are a bunch of Google Tech Talks online which talk about improving web performance. This blog post contains a list of the talks which are most relevant to this topic.