Just to close the loop on this so it appears as answered.
By using index-filter or tree-filter and then applying reverse logic like git ls-tree piped into (multiple) grep -v's piped into xargs for git rm you can indeed remove everything that doesn't match a narrow set of file names/directories. Here is the command I used to split out my particular files:
git filter-branch \
--prune-empty \
--index-filter '
git ls-tree -z -r --name-only --full-tree $GIT_COMMIT \
| grep -z -v "^src/pyfedpkg$" \
| grep -z -v "^src/fedpkg" \
| grep -z -v "^git-changelog" \
| xargs -0 -r git rm --cached -r
' \
-- \
--all
