Since nobody wrote the answer from the comments for several months, I will quickly do so.
There are 2 possible problems/solutions:
id_rsa created with wrong user
Create id_rsa as the jenkins user (see hudson cannot fetch from git repository)
Leave passphrase empty