VSCode remote developement using ssh with passphrase protected ssh-key

Question

I have a remote ssh server on which I want to do remote developement.
My public key is added to the authorized keys on the server and my private key is passphrase pro