Trying to encrypt a text message via command line on OSX Yosomite 10.10.2
Created public .pem
key like this:
ssh-keygen -f ~/.ssh/id_rsa.pub
Your initial solution should work you just have a small typo: To specify key format (PKCS8), the "-m" option is used and not "-t" option (it stand for type of key: dsa, ecdsa, ed25519 or rsa). See ssh-keygen man page.
ssh-keygen -f ~/.ssh/id_rsa.pub -e -m PKCS8 > id_rsa.pem
Then, you could encrypt using this:
openssl rsautl -encrypt -inkey ~/.ssh/id_rsa.pem -pubin -in ~/Desktop/myMessage.txt -out ~/Desktop/encrypted.txt
And, you could decrypt using:
openssl rsautl -decrypt -inkey ~/.ssh/id_rsa -in ~/Desktop/encrypted.txt -out ~/Desktop/decrypted.txt
You could check diffrence between original and decrypted files using text editor or this diff command:
diff ~/Desktop/myMessage.txt ~/Desktop/decrypted.txt
In fact, openssl rsautl -encrypt command expect a public key with "PEM PKCS8 public key" encoding format but ssh-keygen generate a private key in this format and public key in other format adapted to authorized_keys file in ~/.ssh
directory (you could open keys with text editor to see difference between formats).