I had this same problem (where I had done a git clone and I'm using git svn). Here's the command I used to remove the submodule:
git filter-branch --index-filter \
'git rm --cached --ignore-unmatch path/to/the/formerly/misbehaving/module'
I found this wonderful solution on this blog