I want to remove my local repository entirely, but TGitCache.exe keeps an open handle to the directory. I would like to shut it down gracefully, delete the folder and start
Have you seen this?
https://tortoisegit.org/issue/401
It's a bug report on TortoiseGit that seems to be describing your exact problem.
Basically, it seems that TGitCache.exe shouldn't be doing that, and in the newest versions of TortoiseGit it doesn't. Perhaps you might try upgrading your TortoiseGit?