I installed TortoiseGIT using TortoisePink as my SSH client. But I later found out that that was not the correct one to use for my purposes. So, I uninstalled TortoiseGIT an
TortoiseGIT leaves key HKEY_CURRENT_USER\Software\TortoiseGit
in registry after uninstall.
I deleted it and was able to pick SSH client on next install.
I was able to circumvent this problem by going to the TortoiseGIT settings page. Under "network" I noticed that the "TortoisePink.exe" file was still selected. So, I just changed this to "ssh.exe" without a path and viola! My issue is fixed.
I will add this here as I didn't want to reinstall TortoiseGit just for this and the other solutions did not help.
If you want to switch TortoiseGit to OpenSSH, simply run this at the command prompt:
tortoisegitproc /command:firststart