What are some edge overlap minimization techniques when laying out a graph? (Preferably related to GraphViz) Also are there any existing software that can layout a graph in
The following open source Java library has a couple of algorithms which may help in laying out planar graphs. http://open.trickl.com/trickl-graph/index.html
In particular, the following classes provide analytic solutions to the problem:
ChrobakPayneLayout (based on the Boost C++ implementation by Aaron Windsor) http://www.boost.org/doc/libs/1_37_0/libs/graph/doc/straight_line_drawing.html
FoldFreeLayout (based on Anchor-Free Distributed Localization in Sensor Networks * Nissanka B. Priyantha, Hari Balakrishnan, Erik Demaine, and Seth Teller)
What you might want to do is use something like this as the first "attempt" which ensures no overlaps, although may not look great. Then you can apply a force-directed algorithm to space out the nodes more fairly.
Unfortunately, the library has only just been released so is short on documentation. It might however be useful by providing some actual code rather than just theory.
For general graphs, the problem of a determining a planar layout of a graph with least edges crossing (the Crossing Number) is NP-hard. So some heuristic methods are used (like the Force based layout algorithms).
The page below briefly describes the graphviz algorithms and suggests some ways to use them for benefit. It also has links to the pdfs which should contain more information about the algorithms:
http://rss.acs.unt.edu/Rdoc/library/Rgraphviz/html/GraphvizLayouts.html
Hope that helps.