OpenSSL unable to load Public Key

Question

Trying to encrypt a text message via command line on OSX Yosomite 10.10.2

Created public .pem key like this:

ssh-keygen -f ~/.ssh/id_rsa.pub         


        

Answer 1

I faced this problem also and think a good hint is here:

How can I transform between the two styles of public key format, one "BEGIN RSA PUBLIC KEY", the other is "BEGIN PUBLIC KEY"

It seems that the OpenSSL encryption command wants a SSL public key instead of a RSA public key.

We now know enough to tweak the example to make it work. A SSL public key can be generated from a RSA public key with

openssl rsa -in id_rsa.pem -RSAPublicKey_in -pubout > id_pub.pem

It is then possible to do the encryption step with

openssl rsautl -encrypt -inkey id_pub.pem -pubin -in ~/Desktop/myMessage.txt -out ~/Desktop/encrypted.txt

The default OpenSSL command in MacOSX Yosemite as of this writing appears to be 0.9.8zg. The rsa command in this version does not support the capability to run the first command above. I worked around this by installing OpenSSL 1.0.1p.

Answer 2

Your initial solution should work you just have a small typo: To specify key format (PKCS8), the "-m" option is used and not "-t" option (it stand for type of key: dsa, ecdsa, ed25519 or rsa). See ssh-keygen man page.

ssh-keygen -f ~/.ssh/id_rsa.pub -e -m PKCS8 > id_rsa.pem

Then, you could encrypt using this:

openssl rsautl -encrypt -inkey ~/.ssh/id_rsa.pem -pubin -in ~/Desktop/myMessage.txt -out ~/Desktop/encrypted.txt

And, you could decrypt using:

openssl rsautl -decrypt -inkey ~/.ssh/id_rsa -in ~/Desktop/encrypted.txt -out ~/Desktop/decrypted.txt

You could check diffrence between original and decrypted files using text editor or this diff command:

diff ~/Desktop/myMessage.txt ~/Desktop/decrypted.txt

In fact, openssl rsautl -encrypt command expect a public key with "PEM PKCS8 public key" encoding format but ssh-keygen generate a private key in this format and public key in other format adapted to authorized_keys file in ~/.ssh directory (you could open keys with text editor to see difference between formats).

Answer 3

Still don't know what went wrong in my question but found a solution:

1) Generate RSA key:

$ openssl genrsa -out key.pem 1024 
$ openssl rsa -in key.pem -text -noout 

2) Save public key in pub.pem file:

$ openssl rsa -in key.pem -pubout -out pub.pem 
$ openssl rsa -in pub.pem -pubin -text -noout 

3) Encrypt some data:

$ echo test test test > file.txt 
$ openssl rsautl -encrypt -inkey pub.pem -pubin -in file.txt -out file.bin 

4) Decrypt encrypted data:

$ openssl rsautl -decrypt -inkey key.pem -in file.bin 

It works like a charm

Thanks to Marek Marcola for providing the information http://openssl.6102.n7.nabble.com/Re-Can-I-use-my-own-keys-with-openssl-for-RSA-enc-dec-td12506.html

Answer 4

I had same problem when I was extracting public key from certificate.

openssl x509 -pubkey -noout -in cert.crt > pubKey.pem

Afterwards, I wanted to print information about key with command below.

openssl rsa -text -pubin -in pubKey.pem

And gets an error: unable to load Public Key

Solution

I opened pubKey.pem in notepad++ and in the Encoding menu was UCS-2 LE BOM selected. So I changed it to UTF-8 encoding. Size of pubKey.pem was half of the original one after changing encoding. Then it works like charm.

Tested in Windows and powershell