Here is the lay of the land. Like most people I have my domain object and I have my view models. I love the idea of using view models, as it allows for models to be created spec
This may not be appropriate, but what if you just moved your validation rules/annotations from your Models to your ViewModels? In a few of the projects I've been on, we've chosen to prevent the View from accessing anything but information exposed through its corresponding ViewModel. Since all data interaction would be performed through the ViewModel, there wouldn't be a need to have validation on your Model objects.
The counter to this argument is that you could easily duplicate certain validation rules, since different ViewModels might be interfacing with the same Models. In this case, it might make sense to simply declare your Model as a property exposed on your ViewModel. For postbacks, they could accept a Model as their parameter, allowing the ModelBinder infrastructure to handle the request. In this case, if ModelState.IsValid is false, you could just reassign the property to your ViewModel before redisplaying the View.
I would recommend moving your annotations to your ViewModels. It makes sense since a lot of Views are a) the result of composition of several models or b) a subset of model data.