Linux rm delete all files starting with an ! or $ recursively

Question

I have compiled code in various directories which start with an ! or $. I want to remove this from an folder in under a certain directory. I did a cd to the folder. This fold