RSA_public_decrypt and MS Crypto API equivalent

I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's RSA_private_encrypt function.

For Mac OX X I use RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

void cr_license_init(const char* lic) {
    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    BIO* b64 = BIO_new(BIO_f_base64());
    BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
    licIn = BIO_push(b64, licIn);

    if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

        const unsigned char* key_data = license_pub_der;
        RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

        if(r != NULL) {
            if(__cr_license_data_ != NULL) {
                free((void*)__cr_license_data_);
            }
            __cr_license_data_ = malloc(CR_LIC_LEN);
            if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
    (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) {
                free((void*)__cr_license_data_);
                __cr_license_data_ = NULL;
            }
            RSA_free(r);
        }
    }
    BIO_free_all(licIn);
}

This part of code on windows works well, so I assume public key is not an issue.

__cr_license_ = lic;
unsigned char lic_encoded[CR_LIC_LEN];

DWORD dwSize;
if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
HCRYPTPROV hProv;
if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
    PCERT_PUBLIC_KEY_INFO pki = NULL;
    DWORD dwKeySize;
    if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
        HCRYPTKEY hKey = 0;
        if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.

Unless this is so obvious that you've tried but omitted listing it or I misunderstand your question otherwise, I think you should be using CryptDecrypt to decrypt the license, not the functions you mention in the question. Note that since you seem to be using OpenSSL with PKCS#1 v1.5 padding and CryptoAPI does not seem to support that (haven't tested, but specs only list PKCS#1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and verify and remove the PKCS#1 v1.5 padding manually after decryption.

OpenSSL exports keys with extra header which is not expected by CryptoAPI.

Header for private key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   630|      3| SEQUENCE : 
     4|     1|      1|    INTEGER : 0
     7|    13|      1|    SEQUENCE : 
     9|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    20|     0|      1|       NULL : 
    22|   608|      3|    OCTET STRING : 
                             ... actual key data go here ...

Header for public key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   159|      2| SEQUENCE : 
     3|    13|      1|    SEQUENCE : 
     5|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    16|     0|      1|       NULL : 
    18|   141|      2|    BIT STRING UnusedBits:0 : 
                              ... actual key data go here ...

These headers are what causes CryptDecodeObjectEx to choke. It expects RAW key data, without any header.

So, basically, you need:

(Optional) Convert .PEM to .DER with CryptStringToBinary.
Check if DER starts with the above mentioned headers. For that you need to read ASN.1-encoded data.
(Optional) Skip the above mentioned header and seek directly to key's data (starts with SEQUENCE which includes 2 INTEGER for public key or 9 INTEGER for private key).
Feed the result to CryptDecodeObjectEx(X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB/PKCS_RSA_PRIVATE_KEY).
Import keys with CryptImportKey.

来源：https://stackoverflow.com/questions/14527898/rsa-public-decrypt-and-ms-crypto-api-equivalent

标签

windows

cryptography

OpenSSL

rsa

cryptoapi