No, this is possible in with the git command line and double quotes. One easy fix is putting a space after !
.
git commit -m "First Commit! "
Another way to get around this is by using git gui
or just git commit
and then specifying the message in the editor that opens.
The limitation is clearly a bash limitation and not a problem of git. You can avoid it using single quotes:
git commit -m 'First Commit!'