When I try to uninstall pandas from my conda virtual env, I see that it tries to uninstall more packages as well:
pandas
conda
$ conda uninstall
You can use conda remove --force.
conda remove --force
The documentation says:
--force Forces removal of a package without removing packages that depend on it. Using this option will usually leave your environment in a broken and inconsistent state