My wordpress site is a bit heavy to download. On the frontend, its including jquery unnecessarily. In my firebug it looks like:
jquery.js?ver=1.3.2
>
I was able to shave 2.2 seconds off my "Events" page load speed by de-registering jQuery. jQuery is a nice to have but in my opinion page speed is so much more important.
You'll never have a user hang around for more than 5 seconds so if jQuery is causing you performance problems, then I say get rid.