Here is my setup:
To summarise what must be done on the Jenkins server:
# 1. Create the folder containing the SSH keys if necessary
if [ ! -e ~jenkins/.ssh ]; then mkdir ~jenkins/.ssh; fi
cd ~jenkins/.ssh/
# 2. Create the SSH pair of keys
# The comment will help to identify the SSH key on target systems
ssh-keygen -C "jenkins" -f ~jenkins/.ssh/id_rsa -P ""
# 3. Assign the proper access rights
chown -R jenkins ~jenkins/.ssh/
chmod 700 ~jenkins/.ssh
chmod 600 ~jenkins/.ssh/*
Remember:
ssh -vvv git@github.com
(change address according to your git server)