To answer the question directly: yes, it would be wrong because suppressing certainn subclasses from being passed in would violate Liskov Substitution Principle
The fact that you are needing to do that is a strong smell that refractory may be in order...