Ok, so I am stumped on this issue. I have seen a lot of things that are supposed to resolve this issue, but I am not getting a resolution that can fulfill my requirements.
Create a RouteConstraint, that checks if the controller exists (in fact, scans all possible controllers in beginning and caches it, you know what i mean) and add it to your standard route.
so,when you then have a request to a bad controller, it is not accepted by the normal Route. but then accepted by the catchall -> 404 route...