Conventionally in R one can use metacharacters in a regex with two slashes, e.g. ( becomes \\(, but I find the same isn\'t true for square brackets.
mystring
I would sidestep [ab] syntax and use (a|b). Besides working, it may also be more readable:
[ab]
(a|b)
gsub("(\\[|\\]|\\$)","",mystring)