The documentation covers the similar case of purging a file from history:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD
Since you are deleting a whole directory, add the -r flag to git rm:
git filter-branch --index-filter \
'git rm -r --cached --ignore-unmatch path/to/directory' HEAD
Note that this operation will take several minutes on larger repositories.
More importantly, it will make a new repository with distinct history and checksums. If you previously published your repository, the history of the new one will not be compatible with the history others have pulled.
