Went through a lot of questions, but nothing seems to be solving my issue. Or to be more precise I am not sure if I am doing the whole thing correctly. So here it is:
Ha
I think the .ssh directory should live in /var/lib/jenkins.
Works for me