Here is my setup:
I got around this problem by simply leaving the passphrase empty when creating the keys.