I think it depends on your focus. A few years ago I purchased the set of Art of Computer Programming by Donald Knuth. After looking at the books I realized pretty much everything is calculus proofs. If you're interested in developing your own generic algorithms and proofs for them, then I recommend being able to understand the above books since its what you'd be dealing with in that world. On the other hand if you only want/need to use various sorting/searching/tree/etc... routines then big O notation at a minimum, boolean math, and general algebra will be fine. If you're dealing with 3D then geometry and trig as well.
I tend to be more on the using side than making proofs, and while I'd like to think I've done some clever things over the years I've never sat down and developed a new sorting routine. The best advice I can give is learn what you need for your field, but expose yourself to higher levels so you know it exists and how much more there is to learn, you won't get much growth otherwise.
