IdylFin might be just what you are looking for -- inspired by quantmod, it contains some blazing fast, numerically stable statistical methods, an API to Yahoo Finance, as well as some portfolio optimization methods and it's being extended actively.
Disclaimer: I am the author of the library, and it would knock my socks off if you used it ;).