What would be the best approach to implement math formula rendering in a Cocoa application?
There is a utility app included with MacOS X called Grapher that do
As I describe in my answer to this similar question, I built a system for doing this in my iPhone application (which also works on the Mac, but I simply haven't completed that version of the application yet).
I used Core Animation layers to represent the hierarchy of the equation, with Quartz-drawn elements for the various operation types. This let me animate the typesetting in a relatively lightweight manner, which is smooth on even the first-generation iPhone hardware.
Like with the evaluation of arbitrary equations, splitting the layout into hierarchical layers makes the layout calculations pretty straightforward and flexible. However, it still took me months to get everything just right.
If you want to use an existing solution, you might look to Matt Gallagher's excellent open source project Magic Number Machine, which at least one developer has used as the basis for a commercial Mac calculator with equation typesetting.