How to decrypt file in Java encrypted with openssl command using AES?

Question

I need to decrypt in JAVA a file encrypted in UNIX with the following command:

openssl aes-256-cbc -a -salt -in password.txt -out password.txt.enc
mypass
myp         


        

Answer 1

Below are OpenSSLPBEInputStream and OpenSSLPBEOutputStream which can be used to encrypt/decrypt arbitrary streams of bytes in a way that is compatible with OpenSSL.

Example usage:

    // The original clear text bytes
    byte[] originalBytes = ...

    // Encrypt these bytes
    char[] pwd = "thePassword".toCharArray();
    ByteArrayOutputStream byteOS = new ByteArrayOutputStream();
    OpenSSLPBEOutputStream encOS = new OpenSSLPBEOutputStream(byteOS, ALGORITHM, 1, pwd);
    encOS.write(originalBytes);
    encOS.flush();
    byte[] encryptedBytes = byteOS.toByteArray();

    // Decrypt the bytes
    ByteArrayInputStream byteIS = new ByteArrayInputStream(encryptedBytes);
    OpenSSLPBEInputStream encIS = new OpenSSLPBEInputStream(byteIS, ALGORITHM, 1, pwd);

Where ALGORITHM (using just JDK classes) can be: "PBEWithMD5AndDES", "PBEWithMD5AndTripleDES", "PBEWithSHA1AndDESede", "PBEWithSHA1AndRC2_40".

To handle "openssl aes-256-cbc -a -salt -in password.txt -out password.txt.enc" of the original poster, add bouncey castle to the classpath, and use algorthm= "PBEWITHMD5AND256BITAES-CBC-OPENSSL".

/* Add BC provider, and fail fast if BC provider is not in classpath for some reason */
Security.addProvider(new BouncyCastleProvider());

The dependency:

    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcprov-jdk16</artifactId>
        <version>1.44</version>
    </dependency>

The input stream:

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import java.io.IOException;
import java.io.InputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.spec.InvalidKeySpecException;

public class OpenSSLPBEInputStream extends InputStream {

    private final static int READ_BLOCK_SIZE = 64 * 1024;

    private final Cipher cipher;
    private final InputStream inStream;
    private final byte[] bufferCipher = new byte[READ_BLOCK_SIZE];

    private byte[] bufferClear = null;

    private int index = Integer.MAX_VALUE;
    private int maxIndex = 0;

    public OpenSSLPBEInputStream(final InputStream streamIn, String algIn, int iterationCount, char[] password)
            throws IOException {
        this.inStream = streamIn;
        try {
            byte[] salt = readSalt();
            cipher = OpenSSLPBECommon.initializeCipher(password, salt, Cipher.DECRYPT_MODE, algIn, iterationCount);
        } catch (InvalidKeySpecException | NoSuchPaddingException | NoSuchAlgorithmException | InvalidKeyException | InvalidAlgorithmParameterException e) {
            throw new IOException(e);
        }
    }

    @Override
    public int available() throws IOException {
        return inStream.available();
    }

    @Override
    public int read() throws IOException {

        if (index > maxIndex) {
            index = 0;
            int read = inStream.read(bufferCipher);
            if (read != -1) {
                bufferClear = cipher.update(bufferCipher, 0, read);
            }
            if (read == -1 || bufferClear == null || bufferClear.length == 0) {
                try {
                    bufferClear = cipher.doFinal();
                } catch (IllegalBlockSizeException | BadPaddingException e) {
                    bufferClear = null;
                }
            }
            if (bufferClear == null || bufferClear.length == 0) {
                return -1;
            }
            maxIndex = bufferClear.length - 1;
        }
        return bufferClear[index++] & 0xff;

    }

    private byte[] readSalt() throws IOException {

        byte[] headerBytes = new byte[OpenSSLPBECommon.OPENSSL_HEADER_STRING.length()];
        inStream.read(headerBytes);
        String headerString = new String(headerBytes, OpenSSLPBECommon.OPENSSL_HEADER_ENCODE);

        if (!OpenSSLPBECommon.OPENSSL_HEADER_STRING.equals(headerString)) {
            throw new IOException("unexpected file header " + headerString);
        }

        byte[] salt = new byte[OpenSSLPBECommon.SALT_SIZE_BYTES];
        inStream.read(salt);

        return salt;
    }

}

The output stream:

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import java.io.IOException;
import java.io.OutputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.InvalidKeySpecException;

public class OpenSSLPBEOutputStream extends OutputStream {

private static final int BUFFER_SIZE = 5 * 1024 * 1024;

private final Cipher cipher;
private final OutputStream outStream;
private final byte[] buffer = new byte[BUFFER_SIZE];
private int bufferIndex = 0;

public OpenSSLPBEOutputStream(final OutputStream outputStream, String algIn, int iterationCount,
                              char[] password) throws IOException {
    outStream = outputStream;
    try {
        /* Create and use a random SALT for each instance of this output stream. */
        byte[] salt = new byte[PBECommon.SALT_SIZE_BYTES];
        new SecureRandom().nextBytes(salt);
        cipher = OpenSSLPBECommon.initializeCipher(password, salt, Cipher.ENCRYPT_MODE, algIn, iterationCount);
        /* Write header */
        writeHeader(salt);
    } catch (InvalidKeySpecException | NoSuchPaddingException | NoSuchAlgorithmException | InvalidKeyException | InvalidAlgorithmParameterException e) {
        throw new IOException(e);
    }
}

@Override
public void write(int b) throws IOException {
    buffer[bufferIndex] = (byte) b;
    bufferIndex++;
    if (bufferIndex == BUFFER_SIZE) {
        byte[] result = cipher.update(buffer, 0, bufferIndex);
        outStream.write(result);
        bufferIndex = 0;
    }
}

@Override
public void flush() throws IOException {
    if (bufferIndex > 0) {
        byte[] result;
        try {
            result = cipher.doFinal(buffer, 0, bufferIndex);
            outStream.write(result);
        } catch (IllegalBlockSizeException | BadPaddingException e) {
            throw new IOException(e);
        }
        bufferIndex = 0;
    }
}

@Override
public void close() throws IOException {
    flush();
    outStream.close();
}

private void writeHeader(byte[] salt) throws IOException {
    outStream.write(OpenSSLPBECommon.OPENSSL_HEADER_STRING.getBytes(OpenSSLPBECommon.OPENSSL_HEADER_ENCODE));
    outStream.write(salt);
}

}

Small common class:

import javax.crypto.Cipher;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.spec.InvalidKeySpecException;

class OpenSSLPBECommon {

protected static final int SALT_SIZE_BYTES = 8;
protected static final String OPENSSL_HEADER_STRING = "Salted__";
protected static final String OPENSSL_HEADER_ENCODE = "ASCII";

protected static Cipher initializeCipher(char[] password, byte[] salt, int cipherMode,
                                         final String algorithm, int iterationCount) throws NoSuchAlgorithmException, InvalidKeySpecException,
        InvalidKeyException, NoSuchPaddingException, InvalidAlgorithmParameterException {

    PBEKeySpec keySpec = new PBEKeySpec(password);
    SecretKeyFactory factory = SecretKeyFactory.getInstance(algorithm);
    SecretKey key = factory.generateSecret(keySpec);

    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(cipherMode, key, new PBEParameterSpec(salt, iterationCount));

    return cipher;
}

}

Answer 2

In Kotlin:

package io.matthewnelson.java_crypto

import java.util.*
import javax.crypto.Cipher
import javax.crypto.SecretKeyFactory
import javax.crypto.spec.IvParameterSpec
import javax.crypto.spec.PBEKeySpec
import javax.crypto.spec.SecretKeySpec

class OpenSSL {

    /**
     * Will decrypt a string value encrypted by OpenSSL v 1.1.1+ using the following cmds from terminal:
     *
     *   echo "Hello World!" | openssl aes-256-cbc -e -a -p -salt -pbkdf2 -iter 15739 -k qk4aX-EfMUa-g4HdF-fjfkU-bbLNx-15739
     *
     * Terminal output:
     *   salt=CC73B7D29FE59CE1
     *   key=31706F84185EA4B5E8E040F2C813F79722F22996B48B82FF98174F887A9B9993
     *   iv =1420310D41FD7F48E5D8722B9AC1C8DD
     *   U2FsdGVkX1/Mc7fSn+Wc4XLwDsmLdR8O7K3bFPpCglA=
     * */
    fun decrypt_AES256CBC_PBKDF2_HMAC_SHA256(
        password: String,
        hashIterations: Int,
        encryptedString: String
    ): String {
        val encryptedBytes = Base64.getDecoder().decode(encryptedString)

        // Salt is bytes 8 - 15
        val salt = encryptedBytes.copyOfRange(8, 16)
//        println("Salt: ${salt.joinToString("") { "%02X".format(it) }}")

        // Derive 48 byte key
        val keySpec = PBEKeySpec(password.toCharArray(), salt, hashIterations, 48 * 8)
        val keyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256")
        val secretKey = keyFactory.generateSecret(keySpec)

        // Decryption Key is bytes 0 - 31 of the derived key
        val key = secretKey.encoded.copyOfRange(0, 32)
//        println("Key: ${key.joinToString("") { "%02X".format(it) }}")

        // Input Vector is bytes 32 - 47 of the derived key
        val iv = secretKey.encoded.copyOfRange(32, 48)
//        println("IV: ${iv.joinToString("") { "%02X".format(it) }}")

        // Cipher Text is bytes 16 - end of the encrypted bytes
        val cipherText = encryptedBytes.copyOfRange(16, encryptedBytes.lastIndex + 1)

        // Decrypt the Cipher Text and manually remove padding after
        val cipher = Cipher.getInstance("AES/CBC/NoPadding")
        cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), IvParameterSpec(iv))
        val decrypted = cipher.doFinal(cipherText)
//        println("Decrypted: ${decrypted.joinToString("") { "%02X".format(it) }}")

        // Last byte of the decrypted text is the number of padding bytes needed to remove
        val plaintext = decrypted.copyOfRange(0, decrypted.lastIndex + 1 - decrypted.last().toInt())

        return plaintext.toString(Charsets.UTF_8)
    }
}

Answer 3

Don't use ase-128-cbc, use ase-128-ecb.

only take first 16 bytes as key because key is 128 bits

hash output is printed in hex, which every 2 chars presents a byte value

hashpwd=echo -n $password| openssl sha1 | sed 's#.*=\\s*##g' | cut -c 1-32

openssl enc -aes-128-ecb -salt -in -out -K $hashpwd

Java Code is here:

import sun.misc.BASE64Decoder;
import sun.misc.BASE64Encoder;

import javax.crypto.Cipher;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Arrays;


    //openssl enc -nosalt -aes-128-ecb
    // -in <input file>
    // -out <output file>
    // -K <16 bytes in hex, for example : "abc" can be hashed in SHA-1, the first 16 bytes in hex is a9993e364706816aba3e25717850c26c>
    private final static String TRANSFORMATION = "AES"; // use aes-128-ecb in openssl

public static byte[] encrypt(String passcode, byte[] data) throws CryptographicException {
        try {
            Cipher cipher = Cipher.getInstance(TRANSFORMATION);
            cipher.init(Cipher.ENCRYPT_MODE, genKeySpec(passcode));
            return cipher.doFinal(data);
        } catch (Exception ex) {
            throw new CryptographicException("Error encrypting", ex);
        }
    }


    public static String encryptWithBase64(String passcode, byte[] data) throws CryptographicException {
        return new BASE64Encoder().encode(encrypt(passcode, data));
    }

    public static byte[] decrypt(String passcode, byte[] data) throws CryptographicException {
        try {
            Cipher dcipher = Cipher.getInstance(TRANSFORMATION);
            dcipher.init(Cipher.DECRYPT_MODE, genKeySpec(passcode));
            return dcipher.doFinal(data);
        } catch (Exception e) {
            throw new CryptographicException("Error decrypting", e);
        }
    }


    public static byte[] decryptWithBase64(String passcode, String encrptedStr) throws CryptographicException {
        try {
            return decrypt(passcode, new BASE64Decoder().decodeBuffer(encrptedStr));
        } catch (Exception e) {
            throw new CryptographicException("Error decrypting", e);
        }
    }

    public static SecretKeySpec genKeySpec(String passcode) throws UnsupportedEncodingException, NoSuchAlgorithmException {
        byte[] key = passcode.getBytes("UTF-8");
        MessageDigest sha = MessageDigest.getInstance("SHA-1");
        key = sha.digest(key);
        key = Arrays.copyOf(key, 16); // use only first 128 bit
        return new SecretKeySpec(key, TRANSFORMATION);
    }

Tested and passed in jdk6 and jdk8.

Answer 4

OpenSSL generally uses its own password based key derivation method, specified in EVP_BytesToKey, please see the code below. Furthermore, it implicitly encodes the ciphertext as base 64 over multiple lines, which would be required to send it within the body of a mail message.

So the result is, in pseudocode:

salt = random(8)
keyAndIV = BytesToKey(password, salt, 48)
key = keyAndIV[0..31]
iv = keyAndIV[32..47]
ct = AES-256-CBC-encrypt(key, iv, plaintext)
res = base64MimeEncode("Salted__" | salt | ct))

and the decryption therefore is:

(salt, ct) = base64MimeDecode(res)
key = keyAndIV[0..31]
iv = keyAndIV[32..47]
pt = AES-256-CBC-decrypt(key, iv, plaintext)

which can be implemented in Java like this:

import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;
import java.util.Arrays;
import java.util.List;

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

import org.bouncycastle.util.encoders.Base64;

public class OpenSSLDecryptor {
    private static final Charset ASCII = Charset.forName("ASCII");
    private static final int INDEX_KEY = 0;
    private static final int INDEX_IV = 1;
    private static final int ITERATIONS = 1;

    private static final int ARG_INDEX_FILENAME = 0;
    private static final int ARG_INDEX_PASSWORD = 1;

    private static final int SALT_OFFSET = 8;
    private static final int SALT_SIZE = 8;
    private static final int CIPHERTEXT_OFFSET = SALT_OFFSET + SALT_SIZE;

    private static final int KEY_SIZE_BITS = 256;

    /**
     * Thanks go to Ola Bini for releasing this source on his blog.
     * The source was obtained from <a href="http://olabini.com/blog/tag/evp_bytestokey/">here</a> .
     */
    public static byte[][] EVP_BytesToKey(int key_len, int iv_len, MessageDigest md,
            byte[] salt, byte[] data, int count) {
        byte[][] both = new byte[2][];
        byte[] key = new byte[key_len];
        int key_ix = 0;
        byte[] iv = new byte[iv_len];
        int iv_ix = 0;
        both[0] = key;
        both[1] = iv;
        byte[] md_buf = null;
        int nkey = key_len;
        int niv = iv_len;
        int i = 0;
        if (data == null) {
            return both;
        }
        int addmd = 0;
        for (;;) {
            md.reset();
            if (addmd++ > 0) {
                md.update(md_buf);
            }
            md.update(data);
            if (null != salt) {
                md.update(salt, 0, 8);
            }
            md_buf = md.digest();
            for (i = 1; i < count; i++) {
                md.reset();
                md.update(md_buf);
                md_buf = md.digest();
            }
            i = 0;
            if (nkey > 0) {
                for (;;) {
                    if (nkey == 0)
                        break;
                    if (i == md_buf.length)
                        break;
                    key[key_ix++] = md_buf[i];
                    nkey--;
                    i++;
                }
            }
            if (niv > 0 && i != md_buf.length) {
                for (;;) {
                    if (niv == 0)
                        break;
                    if (i == md_buf.length)
                        break;
                    iv[iv_ix++] = md_buf[i];
                    niv--;
                    i++;
                }
            }
            if (nkey == 0 && niv == 0) {
                break;
            }
        }
        for (i = 0; i < md_buf.length; i++) {
            md_buf[i] = 0;
        }
        return both;
    }


    public static void main(String[] args) {
        try {
            // --- read base 64 encoded file ---

            File f = new File(args[ARG_INDEX_FILENAME]);
            List<String> lines = Files.readAllLines(f.toPath(), ASCII);
            StringBuilder sb = new StringBuilder();
            for (String line : lines) {
                sb.append(line.trim());
            }
            String dataBase64 = sb.toString();
            byte[] headerSaltAndCipherText = Base64.decode(dataBase64);

            // --- extract salt & encrypted ---

            // header is "Salted__", ASCII encoded, if salt is being used (the default)
            byte[] salt = Arrays.copyOfRange(
                    headerSaltAndCipherText, SALT_OFFSET, SALT_OFFSET + SALT_SIZE);
            byte[] encrypted = Arrays.copyOfRange(
                    headerSaltAndCipherText, CIPHERTEXT_OFFSET, headerSaltAndCipherText.length);

            // --- specify cipher and digest for EVP_BytesToKey method ---

            Cipher aesCBC = Cipher.getInstance("AES/CBC/PKCS5Padding");
            MessageDigest md5 = MessageDigest.getInstance("MD5");

            // --- create key and IV  ---

            // the IV is useless, OpenSSL might as well have use zero's
            final byte[][] keyAndIV = EVP_BytesToKey(
                    KEY_SIZE_BITS / Byte.SIZE,
                    aesCBC.getBlockSize(),
                    md5,
                    salt,
                    args[ARG_INDEX_PASSWORD].getBytes(ASCII),
                    ITERATIONS);
            SecretKeySpec key = new SecretKeySpec(keyAndIV[INDEX_KEY], "AES");
            IvParameterSpec iv = new IvParameterSpec(keyAndIV[INDEX_IV]);

            // --- initialize cipher instance and decrypt ---

            aesCBC.init(Cipher.DECRYPT_MODE, key, iv);
            byte[] decrypted = aesCBC.doFinal(encrypted);

            String answer = new String(decrypted, ASCII);
            System.out.println(answer);
        } catch (BadPaddingException e) {
            // AKA "something went wrong"
            throw new IllegalStateException(
                    "Bad password, algorithm, mode or padding;" +
                    " no salt, wrong number of iterations or corrupted ciphertext.");
        } catch (IllegalBlockSizeException e) {
            throw new IllegalStateException(
                    "Bad algorithm, mode or corrupted (resized) ciphertext.");
        } catch (GeneralSecurityException e) {
            throw new IllegalStateException(e);
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
    }        
}

Beware that the code specifies ASCII as character set. The character set used may differ for your application / terminal / OS.

---

In general you should force OpenSSL to use the NIST approved PBKDF2 algorithm, as using the OpenSSL key derivation method - with an iteration count of 1 - is insecure. This may force you to use a different solution than OpenSSL. Note that password based encryption is inherently rather insecure - passwords are much less secure than randomly generated symmetric keys.

---

OpenSSL 1.1.0c changed the digest algorithm used in some internal components. Formerly, MD5 was used, and 1.1.0 switched to SHA256. Be careful the change is not affecting you in both EVP_BytesToKey and commands like openssl enc.

It's probably best to explicitly specify the digest in the command line interface (e.g. -md md5 for backwards compatibility or sha-256 for forwards compatibility) for the and make sure that the Java code uses the same digest algorithm ("MD5" or "SHA-256" including the dash). Also see the information in this answer.