RSA_public_decrypt and MS Crypto API equivalent

问题

I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's RSA_private_encrypt function.

For Mac OX X I use RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

void cr_license_init(const char* lic) {
    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    BIO* b64 = BIO_new(BIO_f_base64());
    BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
    licIn = BIO_push(b64, licIn);

    if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

        const unsigned char* key_data = license_pub_der;
        RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

        if(r != NULL) {
            if(__cr_license_data_ != NULL) {
                free((void*)__cr_license_data_);
            }
            __cr_license_data_ = malloc(CR_LIC_LEN);
            if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
    (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) {
                free((void*)__cr_license_data_);
                __cr_license_data_ = NULL;
            }
            RSA_free(r);
        }
    }
    BIO_free_all(licIn);
}

This part of code on windows works well, so I assume public key is not an issue.

__cr_license_ = lic;
unsigned char lic_encoded[CR_LIC_LEN];

DWORD dwSize;
if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
HCRYPTPROV hProv;
if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
    PCERT_PUBLIC_KEY_INFO pki = NULL;
    DWORD dwKeySize;
    if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
        HCRYPTKEY hKey = 0;
        if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

回答1:

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.

回答2:

Unless this is so obvious that you've tried but omitted listing it or I misunderstand your question otherwise, I think you should be using CryptDecrypt to decrypt the license, not the functions you mention in the question. Note that since you seem to be using OpenSSL with PKCS#1 v1.5 padding and CryptoAPI does not seem to support that (haven't tested, but specs only list PKCS#1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and verify and remove the PKCS#1 v1.5 padding manually after decryption.

回答3:

OpenSSL exports keys with extra header which is not expected by CryptoAPI.

Header for private key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   630|      3| SEQUENCE : 
     4|     1|      1|    INTEGER : 0
     7|    13|      1|    SEQUENCE : 
     9|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    20|     0|      1|       NULL : 
    22|   608|      3|    OCTET STRING : 
                             ... actual key data go here ...

Header for public key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   159|      2| SEQUENCE : 
     3|    13|      1|    SEQUENCE : 
     5|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    16|     0|      1|       NULL : 
    18|   141|      2|    BIT STRING UnusedBits:0 : 
                              ... actual key data go here ...

These headers are what causes CryptDecodeObjectEx to choke. It expects RAW key data, without any header.

So, basically, you need:

(Optional) Convert .PEM to .DER with CryptStringToBinary.
Check if DER starts with the above mentioned headers. For that you need to read ASN.1-encoded data.
(Optional) Skip the above mentioned header and seek directly to key's data (starts with SEQUENCE which includes 2 INTEGER for public key or 9 INTEGER for private key).
Feed the result to CryptDecodeObjectEx(X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB/PKCS_RSA_PRIVATE_KEY).
Import keys with CryptImportKey.

来源：https://stackoverflow.com/questions/14527898/rsa-public-decrypt-and-ms-crypto-api-equivalent

标签

windows

cryptography

OpenSSL

rsa

cryptoapi