Github SSH test connection works, but I cannot pull down any of my repos via SSH, only HTTPS

Question

I set up my SSH keys properly, as far as I can tell and when I run the test:

ssh -T git@github.com

I get back:

"Hi {username}! Yo