I have a username on GitHub and have an avatar for that -- easy enough.
The Get a single user V3 API does include my gravatar when I call it with my GitHub id.
Check your author and email information associated with your commits as published inn your GitHub repo.
If they aren't exactly identical to those of your GitHub profile, then your gravatar won't be displayed for those commits.
FYI: GitHub uses your G rated gravatar, so be sure to set that (and not your PG and above).