You can do it using the git filter-branch
command, like this :
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch path_to_file" HEAD
You can find more documentation here http://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository