It's not super Cocoa-friedly, but Graphviz might either do a lot of what you need, or provide a good basis for your own efforts. (That is, the graphviz library can handle all the graph layout logic, and tell you where to put shapes which you then draw using your favorite graphics API. For example, this is how OmniGraffle's automatic layout feature works.)
