In my mind @slugslog had it, but I would change one thing. I would use replace instead so you don't have to set it back.
$scope.scrollTo = function(id) {
var old = $location.hash();
$location.hash(id).replace();
$anchorScroll();
};
Docs Search for "Replace method"