openssl des3 decrypting in java

Question

is there a way to decrypt files that have been encrypted using openssl -des3 enc command. Exactly how does openssl use the password and salt to make the key?

Answer 1

Thank you, Erickson, for your post. It helped me tremendously trying to recreate openssl's password to key and IV routine.

I ended up with something slightly different, probably because I need to decrypt blowfish-encrypted data rather than DES. See below.

Also I've discovered that openssl will stop reading passwords when it encounters bytes 00, 0a, or 0d. Generally I think that openssl only reads password characters between bytes 11 and 127. So for the example below, I have code that precedes this that truncates the password if it contains 00, 0a or 0d.

     /* Compute the key and IV with OpenSSL's non-standard method. */
     final byte[] digest = new byte[32];
     final MessageDigest md5 = MessageDigest.getInstance("MD5");
     md5.update(password, 0);
     // append the salt
     md5.update(salt);
     // run the digest and output 16 bytes to the first 16 bytes to the digest array. Digest is reset
     md5.digest(digest, 0, 16);
     // write the first 16 bytes from the digest array back to the buffer
     md5.update(digest, 0, 16);
     // append the password
     md5.update(password, 0);
     // append the salt
     md5.update(salt);
     // run the digest and output 16 bytes to the last 16 bytes of the digest array
     md5.digest(digest, 16, 16);
     key = Arrays.copyOfRange(digest, 0, 16);
     iv = Arrays.copyOfRange(digest, 16, 24);

This code above can be replaced with 3 lines using org.bouncycastle.crypto.generators.OpenSSLPBEParametersGenerator. It becomes

final OpenSSLPBEParametersGenerator generator = new OpenSSLPBEParametersGenerator();
generator.init(password, salt);
final ParametersWithIV ivParam = (ParametersWithIV)generator.generateDerivedParameters(16, 8);
final KeyParameter keyParameter = (KeyParameter)ivParam.getParameters();

Answer 2

OpenSSL's enc utility uses a non-standard (and low quality) key derivation algorithm for passwords. The following code shows how the enc utility generates the key and initialization vector, given salt and a password. Note that enc stores the "salt" value in the encrypted file when the -salt option is specified (and that is critical for security).

public InputStream decrypt(InputStream is, byte[] password)
  throws GeneralSecurityException, IOException
{
  /* Parse the "salt" value from the stream. */
  byte[] header = new byte[16];
  for (int idx = 0; idx < header.length;) {
    int n = is.read(header, idx, header.length - idx);
    if (n < 0)
      throw new EOFException("File header truncated.");
    idx += n;
  }
  String magic = new String(header, 0, 8, "US-ASCII");
  if (!"Salted__".equals(magic))
    throw new IOException("Expected salt in header.");

  /* Compute the key and IV with OpenSSL's non-standard method. */
  SecretKey secret;
  IvParameterSpec iv;
  byte[] digest = new byte[32];
  try {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(password);
    md5.update(header, 8, 8);
    md5.digest(digest, 0, 16);
    md5.update(digest, 0, 16);
    md5.update(password);
    md5.update(header, 8, 8);
    md5.digest(digest, 16, 16);
    iv = new IvParameterSpec(digest, 24, 8);
    DESedeKeySpec keySpec = new DESedeKeySpec(digest);
    SecretKeyFactory factory = SecretKeyFactory.getInstance("DESede");
    secret = factory.generateSecret(keySpec);
  }
  finally {
    Arrays.fill(digest, (byte) 0);
  }

  /* Initialize the cipher. */
  Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
  cipher.init(Cipher.DECRYPT_MODE, secret, iv);
  return new CipherInputStream(is, cipher);
}

This key and IV generation are described in the EVP_BytesToKey(3) documentation. The enc command uses 1 as the iteration count (which is a bad idea, and noted as a bug in the man page for my version of enc), and MD5 as the digest algorithm—a "broken" algorithm.

It is not clear how a OpenSSL converts text password to bytes. I'm guessing it uses the default platform character encoding. So, if you are stuck with a String password (not good, since it can't be "zero-ized"), you can just call password.getBytes() to convert it to a byte[].

If you can, use something like Java 6's Console or Swing's JPasswordField to get a password. These return an array, so you can "delete" the password from memory when you are done with it: Arrays.fill(password, '\0');