If you want something as sophisticated as I think you are asking for, the only thing that comes to mind that's Java-ecosystem compatible is Clojuratica, and even then you'd have to store the formulae in Mathematica format. (And have a copy of Mathematica.)
