echo '#!'
Basically, with double quotes ("
) aka "weak quoting", Bash does some wacky things with the string, like variable substitution. With single ('
) aka "strong quoting" the string is taken literally.
See e.g. here for a more in-depth explanation of quoting.