openssl des3 decrypting in java

Question

is there a way to decrypt files that have been encrypted using openssl -des3 enc command. Exactly how does openssl use the password and salt to make the key?

Answer 1

OpenSSL's enc utility uses a non-standard (and low quality) key derivation algorithm for passwords. The following code shows how the enc utility generates the key and initialization vector, given salt and a password. Note that enc stores the "salt" value in the encrypted file when the -salt option is specified (and that is critical for security).

public InputStream decrypt(InputStream is, byte[] password)
  throws GeneralSecurityException, IOException
{
  /* Parse the "salt" value from the stream. */
  byte[] header = new byte[16];
  for (int idx = 0; idx < header.length;) {
    int n = is.read(header, idx, header.length - idx);
    if (n < 0)
      throw new EOFException("File header truncated.");
    idx += n;
  }
  String magic = new String(header, 0, 8, "US-ASCII");
  if (!"Salted__".equals(magic))
    throw new IOException("Expected salt in header.");

  /* Compute the key and IV with OpenSSL's non-standard method. */
  SecretKey secret;
  IvParameterSpec iv;
  byte[] digest = new byte[32];
  try {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(password);
    md5.update(header, 8, 8);
    md5.digest(digest, 0, 16);
    md5.update(digest, 0, 16);
    md5.update(password);
    md5.update(header, 8, 8);
    md5.digest(digest, 16, 16);
    iv = new IvParameterSpec(digest, 24, 8);
    DESedeKeySpec keySpec = new DESedeKeySpec(digest);
    SecretKeyFactory factory = SecretKeyFactory.getInstance("DESede");
    secret = factory.generateSecret(keySpec);
  }
  finally {
    Arrays.fill(digest, (byte) 0);
  }

  /* Initialize the cipher. */
  Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
  cipher.init(Cipher.DECRYPT_MODE, secret, iv);
  return new CipherInputStream(is, cipher);
}

This key and IV generation are described in the EVP_BytesToKey(3) documentation. The enc command uses 1 as the iteration count (which is a bad idea, and noted as a bug in the man page for my version of enc), and MD5 as the digest algorithm—a "broken" algorithm.

It is not clear how a OpenSSL converts text password to bytes. I'm guessing it uses the default platform character encoding. So, if you are stuck with a String password (not good, since it can't be "zero-ized"), you can just call password.getBytes() to convert it to a byte[].

If you can, use something like Java 6's Console or Swing's JPasswordField to get a password. These return an array, so you can "delete" the password from memory when you are done with it: Arrays.fill(password, '\0');