Most of times, this seems to be because ssh doesn't pass the right public key to use to gitolite. Cf. "Does Not Appear to be a Git Repository". A comment in this article links to the gitolite author's website, especially to the 'Appendix #2', which explains how to know "which key is which".