OpenSSL and MS CryptoAPI: different digital signatures

Question

I generated X509 certificate with private key using makecert utility

makecert -n "CN=RootCATest" -r -sv RootCATest.pvk RootCATest.cer 
makecert -sk MyKeyName -iv RootCATest.pvk -n "CN=tempCert" -ic RootCATest.cer -sr currentuser -ss my -sky signature —pe 

Then I converted RootCATest.pvk to RootCATest.pem with OpenSSL. And I extracted public key: pubRootCATest.pem

I have small file called 'msg'. And I sign this file using SHA1.

openssl dgst -sha1 -sign c:\RootCATest.pem -out c:\openssl c:\msg

Then I want to obtain the same digital signature using MS CryptoAPI.

Here is my code (Note: this is the code to understand concepts so I don't free allocated memory)

void SwapBytes(BYTE *pv, int n)
{
    BYTE *p = pv;
    int lo, hi;
    for(lo=0, hi=n-1; hi>lo; lo++, hi--)
    {
        BYTE tmp=p[lo];
        p[lo] = p[hi];
        p[hi] = tmp;
    }
}

void sign()
{
    FILE *file;
    BYTE *msg;
    int msg_size;

    HCRYPTPROV hProv;
    HCERTSTORE hStore;
    PCCERT_CONTEXT pCert;
    DWORD dwKeySpec;
    BOOL fCallerFreeProv;
    BYTE  *pSignature;
    DWORD sigLen;

    // Read message bytes from file
    file = fopen("c:\\msg", "r");
    fseek(file, 0, SEEK_END);
    msg_size = ftell(file);
    fseek(file, 0, SEEK_SET);
    msg = new BYTE[msg_size];
    fread(msg, sizeof(BYTE), msg_size, file);
    fclose(file);

    hStore = CertOpenSystemStore(NULL, "My");
    pCert = CryptUIDlgSelectCertificateFromStore(hStore, NULL, NULL, NULL, 0, 0, NULL);
    CryptAcquireCertificatePrivateKey(pCert, CRYPT_ACQUIRE_COMPARE_KEY_FLAG, NULL, &hProv, &dwKeySpec, &fCallerFreeProv);
    PrintCryptoProviderName(hProv); // prints Microsoft Strong Cryptographic Provider

    ALG_ID hashAlgId = CALG_SHA1;
    HCRYPTHASH hHash;
    CryptCreateHash(hProv, CALG_SHA1, 0, 0, &hHash);
    CryptHashData(hHash, msg, msg_size, 0);

    CryptSignHash(hHash, dwKeySpec, NULL, 0, NULL, &sigLen);
    pSignature = new BYTE[sigLen];
    CryptSignHash(hHash, dwKeySpec, NULL, CRYPT_NOHASHOID, pSignature, &sigLen);

    SwapBytes(pSignature, sigLen); // Here i reverse byte order as I read that MS CryptoAPI uses reversed byte order

    // Write signature bytes to file
    file = fopen("c:\\CryptSignHash", "w");
    fwrite(pSignature, sizeof(BYTE), sigLen, file);
    fclose(file);
}

As output I get the signature absolutely different from the signature made by OpenSSL. How can I obtain the same signature?

As I consider there are some moments to pay attention:

• My msg_size is the same as file size. So it is the number of bytes to sign. On some sites I saw recommendations to add a null byte to byte array. Do I really need it in such a case?
• The flag CRYPT_NOHASHOID. Without it I get the signature of size 130 bytes, when the signature made by OpenSSL is 128 bytes. So I think CRYPT_NOHASHOID should be there.
• SwapBytes(...) I tried with it and without it. And in both cases I have signatures absolutely different to OpenSSL signature.

Answer 1

How can I obtain the same signature?

Most digital signature algorithms - including RSA, which I suppose you have used here, are non-deterministic. Try signing the same file twice with the same program, and you will get different outputs.

This means, running the same algorithm twice with the same input will give you different signatures. This is not a problem, as long as the verification algorithm still manages to accept all signatures generated by the signing algorithm (with the fitting key).

This non-determinism is often actually necessary for the security of the signature scheme.

To see if your two signature algorithms are actually compatible, try to verify the OpenSSL signature with the MS Crypto API, and to verify the MS Crypto signature with OpenSSL. (Then modify the file by one byte and check that they don't verify anymore.)