Failure to decrypt an AES encrypted SMIME Message with OpenSSL

问题

I have a two mails (SMIME encrypted) for a single recipient. One mail is encrypted using 3DES, the other one is encrypted using AES 256.

The mails where created using C# EnvelopedCms class.

I can successfully decrypt the 3DES message using

openssl smime -decrypt -in trippledes.eml -inkey keyfile.pem

However, if I try this with the AES encrypted file, OpenSSL outputs some gibberish and Fails with this comment:

Error decrypting PKCS#7 structure 4128:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:.\crypto\evp\evp_enc.c:539:

Thunderbird cannot open the mail either. But Outlook 2010 has no problem opening the message.

What is the best way to troubleshoot the issue? Is there any logging I can turn on?

I've already examined the ASN.1 structure of both mails using the ASN.1 Decoder on http://lapo.it/asn1js/. Both messages look OK to me, so I guess the culprit lies in the encrypted Content itself.

回答1:

I know this is years late, but it might be helpful to others...

I was using the EnvelopedCms very successfully and happily for a few years, exchanging messages with many other implementations. When someone this year decided to require the use of AES, I discovered that at least one Java-based system was failing to work with my messages. (Their error was "Unable to create PKCS #7 MIME content")

I used an ASN info utility to break down what I was sending, and discovered that EnvelopedCms was forcing the KeyEncryptionAlgorithm to RSA-OAEP when the content encryption was set to AES. (If the content was encrypted with anything else, the KeyEncryptionAlgorithm was just plain RSA.)

I could find no documentation or RFCs explaining this behavior, and there does not appear to be any way to change it.

My solution to the problem was to use the BouncyCastle CmsEnvelopedDataGenerator class. So far, it appears to work at least as well as EnvelopedCms, and avoids the RSA-OAEP key encryption issue. Almost a drop-in replacement (other than the certificate class used.)

While I could not find any documentation that specifically said that my recipient's Java libraries could not use the RSA-OAEP algorithm, once I eliminated it, their error was gone, and messages were successfully sent.

Example code using BouncyCastle:

private byte[] CmsEncrypt(byte[] message, string contentEncryptionOid, Org.BouncyCastle.X509.X509Certificate recipCertificate)
{
    var cmsGenerator = new CmsEnvelopedDataGenerator();
    var cmsData = new CmsProcessableByteArray(message);

    cmsGenerator.AddKeyTransRecipient(recipCertificate);

    var cmsEnvelope = cmsGenerator.Generate(cmsData, contentEncryptionOid);

    return cmsEnvelope.GetEncoded();
}

来源：https://stackoverflow.com/questions/13917640/failure-to-decrypt-an-aes-encrypted-smime-message-with-openssl