Decrypting an OpenSSL PEM Encoded RSA private key with Java?

Question

I have an encrypted private key and I know the password.

I need to decrypt it using a Java library.

I\'d prefer not to use BouncyCastle though, unless ther

Answer 1

Java code example below shows how to construct the decryption key to obtain the underlying RSA key from an encrypted private key created using the openssl 1.0.x genrsa command; specifically from the following genrsa options that may have been leveraged:

-des encrypt the generated key with DES in cbc mode

-des3 encrypt the generated key with DES in ede cbc mode (168 bit key)

-aes128, -aes192, -aes256 encrypt PEM output with cbc aes

Above options result in encrypted RSA private key of the form ...

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AAA,BBB
...

Where AAA would be one of:

DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC

AND BBB is the hex-encoded IV value

KeyFactory factory = KeyFactory.getInstance("RSA");
KeySpec keySpec = null;
RSAPrivateKey privateKey = null;

Matcher matcher = OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN.matcher(pemContents);
if (matcher.matches())
{
    String encryptionDetails = matcher.group(1).trim(); // e.g. AES-256-CBC,XXXXXXX
    String encryptedKey = matcher.group(2).replaceAll("\\s", ""); // remove tabs / spaces / newlines / carriage return etc

    System.out.println("PEM appears to be OpenSSL Encrypted RSA Private Key; Encryption details : "
        + encryptionDetails + "; Key : " + encryptedKey);

    byte[] encryptedBinaryKey = java.util.Base64.getDecoder().decode(encryptedKey);

    String[] encryptionDetailsParts = encryptionDetails.split(",");
    if (encryptionDetailsParts.length == 2)
    {
        String encryptionAlgorithm = encryptionDetailsParts[0];
        String encryptedAlgorithmParams = encryptionDetailsParts[1]; // i.e. the initialization vector in hex

        byte[] pw = new String(password).getBytes(StandardCharsets.UTF_8);
        byte[] iv = fromHex(encryptedAlgorithmParams);

        MessageDigest digest = MessageDigest.getInstance("MD5");

        // we need to come up with the encryption key
        
        // first round digest based on password and first 8-bytes of IV ..
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round1Digest = digest.digest(); // The digest is reset after this call is made.
        
        // second round digest based on first round digest, password, and first 8-bytes of IV ...
        digest.update(round1Digest);
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round2Digest = digest.digest();

        Cipher cipher = null;
        SecretKey secretKey = null;
        byte[] key = null;
        byte[] pkcs1 = null;

        if ("AES-256-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[32]; // 256 bit key  (block size still 128-bit)
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-192-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-128-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[16]; // 128 bit key
            System.arraycopy(round1Digest, 0, key, 0, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("DES-EDE3-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
            
            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "DESede");
        }
        else if ("DES-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DES/CBC/PKCS5Padding");
            
            key = new byte[8]; // key size of 8 bytes
            System.arraycopy(round1Digest, 0, key, 0, 8);

            secretKey = new SecretKeySpec(key, "DES");
        }

        cipher.init(Cipher.DECRYPT_MODE, secretKey, new IvParameterSpec(iv));

        pkcs1 = cipher.doFinal(encryptedBinaryKey);

        keySpec = pkcs1ParsePrivateKey(pkcs1);

        privateKey = (RSAPrivateKey) factory.generatePrivate(keySpec);
    }
}

The regular expression ...

static final String OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX = "\\s*" 
+ "-----BEGIN RSA PUBLIC KEY-----" + "\\s*"
+ "Proc-Type: 4,ENCRYPTED" + "\\s*"
+ "DEK-Info:" + "\\s*([^\\s]+)" + "\\s+"
+ "([\\s\\S]*)"
+ "-----END RSA PUBLIC KEY-----" + "\\s*";

static final Pattern OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN = Pattern.compile(OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX);

the fromHex(...) method ...

public static byte[] fromHex(String hexString)
{
    byte[] bytes = new byte[hexString.length() / 2];
    for (int i = 0; i < hexString.length(); i += 2)
    {
        bytes[i / 2] = (byte) ((Character.digit(hexString.charAt(i), 16) << 4)
            + Character.digit(hexString.charAt(i + 1), 16));
    }
    return bytes;
}

Answer 2

You need to use a non-standard, OpenSSL method for deriving the decryption key. Then use that to decrypt the PKCS-#1–encoded key—what you are working with is not a PKCS #8 envelope. You'll also need the IV from the header as input to these processes.

It looks something like this:

  static RSAPrivateKey decrypt(String keyDataStr, String ivHex, String password)
    throws GeneralSecurityException
  {
    byte[] pw = password.getBytes(StandardCharsets.UTF_8);
    byte[] iv = h2b(ivHex);
    SecretKey secret = opensslKDF(pw, iv);
    Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
    cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
    byte[] pkcs1 = cipher.doFinal(Base64.getMimeDecoder().decode(keyDataStr));
    /* See note for definition of "decodeRSAPrivatePKCS1" */
    RSAPrivateCrtKeySpec spec = decodeRSAPrivatePKCS1(pkcs1);
    KeyFactory rsa = KeyFactory.getInstance("RSA");
    return (RSAPrivateKey) rsa.generatePrivate(spec);
  }

  private static SecretKey opensslKDF(byte[] pw, byte[] iv)
    throws NoSuchAlgorithmException
  {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(pw);
    md5.update(iv);
    byte[] d0 = md5.digest();
    md5.update(d0);
    md5.update(pw);
    md5.update(iv);
    byte[] d1 = md5.digest();
    byte[] key = new byte[24];
    System.arraycopy(d0, 0, key, 0, 16);
    System.arraycopy(d1, 0, key, 16, 8);
    return new SecretKeySpec(key, "DESede");
  }

  private static byte[] h2b(CharSequence s)
  {
    int len = s.length();
    byte[] b = new byte[len / 2];
    for (int src = 0, dst = 0; src < len; ++dst) {
      int hi = Character.digit(s.charAt(src++), 16);
      int lo = Character.digit(s.charAt(src++), 16);
      b[dst] = (byte) (hi << 4 | lo);
    }
    return b;
  }

This is already a lot of code, so I will link to another answer for the definition of the decodeRSAPrivatePKCS1() method.