Can you help me get my head around openssl public key encryption with rsa.h in c++?

Question

I am trying to get my head around public key encryption using the openssl implementation of rsa in C++. Can you help? So far these are my thoughts (please do correct if nece

Answer 1

Thanks @Caf. Your post helped. However I got

The program '[7056] Encryption2.exe: Native' has exited with code -1073741811 (0xc000000d) for the line

  PEM_read_RSA_PUBKEY(rsa_pkey_file, &rsa_pkey, NULL, NULL)

I changed to

BIO *bio;
X509 *certificate;

bio = BIO_new(BIO_s_mem());
BIO_puts(bio, (const char*)data);
certificate = PEM_read_bio_X509(bio, NULL, NULL, NULL);
EVP_PKEY *pubkey = X509_get_pubkey (certificate);
rsa_pkey = EVP_PKEY_get1_RSA(pubkey);

Where data has the PEM file with only public key. My challenge was to encrypt in C++ and decrypt in java. I transmitted the base64 encoded ek of size eklen (i did not use eklen_n) and decrypted to get the AES key using the RSA private key. Then I decrypted the cipher file using this AES key. It worked fine.

Answer 2

You should actually be using the higher-level "Envelope Encryption" functions from openssl/evp.h, rather than the low-level RSA functions directly. These do most of the work for you and mean you don't have to reinvent the wheel.

In this case, you'd use the EVP_SealInit(), EVP_SealUpdate() and EVP_SealFinal() functions. The corresponding decryption functions are EVP_OpenInit(), EVP_OpenUpdate() and EVP_OpenFinal(). I would suggest using EVP_aes_128_cbc() as the value of the cipher type parameter.

Once you've got the public key loaded into an RSA * handle, you use EVP_PKEY_assign_RSA() to put it into an EVP_PKEY * handle for the EVP functions.

Once you've got this going, to solve the authentication problem I mentioned in my comment, you'll need to established a trusted authority ("Trent"). Trent's public key is known to all users (distributed with the application or similar - just load it from a PEM file). Instead of exchanging bare RSA parameters, Alice and Bob exchange x509 certificates that contain their RSA public keys together with their name, and are signed by Trent. Alice and Bob then each verify the certificate they recieved from the other (using Trent's public key, which they already know), including checking that the associated name is the right one, before continuing the protocol. OpenSSL includes functions for loading and verifying certificates in the x509.h header.

---

Here's an example of how to use EVP_Seal*() to encrypt a file given the recipient's public key. It takes the PEM RSA Public Key file (ie as generated by openssl rsa -pubout) as a command line argument, reads the source data from stdin and writes the encrypted data to stdout. To decrypt, use EVP_Open*() instead, and PEM_read_RSAPrivateKey() to read a private key rather than public key.

It's not really that hard - and certainly less error prone than messing about generating padding, IVs and so on yourself (the Seal function does both the RSA and AES parts of the deal). Anyway, the code:

#include <stdio.h>
#include <stdlib.h>

#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/err.h>

#include <arpa/inet.h> /* For htonl() */

int do_evp_seal(FILE *rsa_pkey_file, FILE *in_file, FILE *out_file)
{
    int retval = 0;
    RSA *rsa_pkey = NULL;
    EVP_PKEY *pkey = EVP_PKEY_new();
    EVP_CIPHER_CTX ctx;
    unsigned char buffer[4096];
    unsigned char buffer_out[4096 + EVP_MAX_IV_LENGTH];
    size_t len;
    int len_out;
    unsigned char *ek;
    int eklen;
    uint32_t eklen_n;
    unsigned char iv[EVP_MAX_IV_LENGTH];

    if (!PEM_read_RSA_PUBKEY(rsa_pkey_file, &rsa_pkey, NULL, NULL))
    {
        fprintf(stderr, "Error loading RSA Public Key File.\n");
        ERR_print_errors_fp(stderr);
        retval = 2;
        goto out;
    }

    if (!EVP_PKEY_assign_RSA(pkey, rsa_pkey))
    {
        fprintf(stderr, "EVP_PKEY_assign_RSA: failed.\n");
        retval = 3;
        goto out;
    }

    EVP_CIPHER_CTX_init(&ctx);
    ek = malloc(EVP_PKEY_size(pkey));

    if (!EVP_SealInit(&ctx, EVP_aes_128_cbc(), &ek, &eklen, iv, &pkey, 1))
    {
        fprintf(stderr, "EVP_SealInit: failed.\n");
        retval = 3;
        goto out_free;
    }

    /* First we write out the encrypted key length, then the encrypted key,
     * then the iv (the IV length is fixed by the cipher we have chosen).
     */

    eklen_n = htonl(eklen);
    if (fwrite(&eklen_n, sizeof eklen_n, 1, out_file) != 1)
    {
        perror("output file");
        retval = 5;
        goto out_free;
    }
    if (fwrite(ek, eklen, 1, out_file) != 1)
    {
        perror("output file");
        retval = 5;
        goto out_free;
    }
    if (fwrite(iv, EVP_CIPHER_iv_length(EVP_aes_128_cbc()), 1, out_file) != 1)
    {
        perror("output file");
        retval = 5;
        goto out_free;
    }

    /* Now we process the input file and write the encrypted data to the
     * output file. */

    while ((len = fread(buffer, 1, sizeof buffer, in_file)) > 0)
    {
        if (!EVP_SealUpdate(&ctx, buffer_out, &len_out, buffer, len))
        {
            fprintf(stderr, "EVP_SealUpdate: failed.\n");
            retval = 3;
            goto out_free;
        }

        if (fwrite(buffer_out, len_out, 1, out_file) != 1)
        {
            perror("output file");
            retval = 5;
            goto out_free;
        }
    }

    if (ferror(in_file))
    {
        perror("input file");
        retval = 4;
        goto out_free;
    }

    if (!EVP_SealFinal(&ctx, buffer_out, &len_out))
    {
        fprintf(stderr, "EVP_SealFinal: failed.\n");
        retval = 3;
        goto out_free;
    }

    if (fwrite(buffer_out, len_out, 1, out_file) != 1)
    {
        perror("output file");
        retval = 5;
        goto out_free;
    }

    out_free:
    EVP_PKEY_free(pkey);
    free(ek);

    out:
    return retval;
}

int main(int argc, char *argv[])
{
    FILE *rsa_pkey_file;
    int rv;

    if (argc < 2)
    {
        fprintf(stderr, "Usage: %s <PEM RSA Public Key File>\n", argv[0]);
        exit(1);
    }

    rsa_pkey_file = fopen(argv[1], "rb");
    if (!rsa_pkey_file)
    {
        perror(argv[1]);
        fprintf(stderr, "Error loading PEM RSA Public Key File.\n");
        exit(2);
    }

    rv = do_evp_seal(rsa_pkey_file, stdin, stdout);

    fclose(rsa_pkey_file);
    return rv;
}

---

The code you've posted illustrates nicely why you should use the higher-level functions - you've fallen into a couple of pitfalls:

• rand() is emphatically not a cryptographically strong random number generator! Generating your symmetric key using rand() is enough to make the entire system completely insecure. (The EVP_*() functions generate the necessary random numbers themselves, using a cryptographically strong RNG, seeded from an appropriate entropy source).

• You are setting the IV for CFB mode to a fixed value (zero). This negates any advantage of using CFB mode in the first place (allowing attackers to trivially perform block-replacement attacks and worse). (The EVP_*() functions generate an appropriate IV for you, when required).

• RSA_PKCS1_OAEP_PADDING should be used if you're defining a new protocol, rather than interoperating with an existing protocol.

---

The corresponding decryption code, for posterity:

#include <stdio.h>
#include <stdlib.h>

#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/err.h>

#include <arpa/inet.h> /* For htonl() */

int do_evp_unseal(FILE *rsa_pkey_file, FILE *in_file, FILE *out_file)
{
    int retval = 0;
    RSA *rsa_pkey = NULL;
    EVP_PKEY *pkey = EVP_PKEY_new();
    EVP_CIPHER_CTX ctx;
    unsigned char buffer[4096];
    unsigned char buffer_out[4096 + EVP_MAX_IV_LENGTH];
    size_t len;
    int len_out;
    unsigned char *ek;
    unsigned int eklen;
    uint32_t eklen_n;
    unsigned char iv[EVP_MAX_IV_LENGTH];

    if (!PEM_read_RSAPrivateKey(rsa_pkey_file, &rsa_pkey, NULL, NULL))
    {
        fprintf(stderr, "Error loading RSA Private Key File.\n");
        ERR_print_errors_fp(stderr);
        retval = 2;
        goto out;
    }

    if (!EVP_PKEY_assign_RSA(pkey, rsa_pkey))
    {
        fprintf(stderr, "EVP_PKEY_assign_RSA: failed.\n");
        retval = 3;
        goto out;
    }

    EVP_CIPHER_CTX_init(&ctx);
    ek = malloc(EVP_PKEY_size(pkey));

    /* First need to fetch the encrypted key length, encrypted key and IV */

    if (fread(&eklen_n, sizeof eklen_n, 1, in_file) != 1)
    {
        perror("input file");
        retval = 4;
        goto out_free;
    }
    eklen = ntohl(eklen_n);
    if (eklen > EVP_PKEY_size(pkey))
    {
        fprintf(stderr, "Bad encrypted key length (%u > %d)\n", eklen,
            EVP_PKEY_size(pkey));
        retval = 4;
        goto out_free;
    }
    if (fread(ek, eklen, 1, in_file) != 1)
    {
        perror("input file");
        retval = 4;
        goto out_free;
    }
    if (fread(iv, EVP_CIPHER_iv_length(EVP_aes_128_cbc()), 1, in_file) != 1)
    {
        perror("input file");
        retval = 4;
        goto out_free;
    }

    if (!EVP_OpenInit(&ctx, EVP_aes_128_cbc(), ek, eklen, iv, pkey))
    {
        fprintf(stderr, "EVP_OpenInit: failed.\n");
        retval = 3;
        goto out_free;
    }

    while ((len = fread(buffer, 1, sizeof buffer, in_file)) > 0)
    {
        if (!EVP_OpenUpdate(&ctx, buffer_out, &len_out, buffer, len))
        {
            fprintf(stderr, "EVP_OpenUpdate: failed.\n");
            retval = 3;
            goto out_free;
        }

        if (fwrite(buffer_out, len_out, 1, out_file) != 1)
        {
            perror("output file");
            retval = 5;
            goto out_free;
        }
    }

    if (ferror(in_file))
    {
        perror("input file");
        retval = 4;
        goto out_free;
    }

    if (!EVP_OpenFinal(&ctx, buffer_out, &len_out))
    {
        fprintf(stderr, "EVP_OpenFinal: failed.\n");
        retval = 3;
        goto out_free;
    }

    if (fwrite(buffer_out, len_out, 1, out_file) != 1)
    {
        perror("output file");
        retval = 5;
        goto out_free;
    }

    out_free:
    EVP_PKEY_free(pkey);
    free(ek);

    out:
    return retval;
}

int main(int argc, char *argv[])
{
    FILE *rsa_pkey_file;
    int rv;

    if (argc < 2)
    {
        fprintf(stderr, "Usage: %s <PEM RSA Private Key File>\n", argv[0]);
        exit(1);
    }

    rsa_pkey_file = fopen(argv[1], "rb");
    if (!rsa_pkey_file)
    {
        perror(argv[1]);
        fprintf(stderr, "Error loading PEM RSA Private Key File.\n");
        exit(2);
    }

    rv = do_evp_unseal(rsa_pkey_file, stdin, stdout);

    fclose(rsa_pkey_file);
    return rv;
}

Answer 3

I write two examples around CAF's code. They are heavily modifed and uses OpenSSL's BIO container for more abstraction.

One example uses a file as input and the other example uses a string buffer. It uses RSA and DES, however you can easily change it from the code. Compile instructions are inside the code. I needed a working example, I hope someone find this useful. I also commented the code. You can get it from here:

Take file as input: https://github.com/farslan/snippets/blob/master/hybrid_file.c

Take string buffer as input: https://github.com/farslan/snippets/blob/master/hybrid_data.c

Answer 4

Actually, no problem, I have just read that basically, the RSA object is a structure that contains both public and private fields. One can extract the public field data and only send that to Bob.

I.e. basically, to extract the public fields from rsa and store each in two different buffers (which are char arrays and can then be sent to Bob), you do:

sprintf(keyBufferA,"%s\r\n",BN_bn2hex(rsa->n));
sprintf(keyBufferB,"%s\r\n",BN_bn2hex(rsa->e));

And then Bob, on the receiving end, reconstructs as follows:

rsa = RSA_new();
BN_hex2bn(&rsa->n, keybufA);
BN_hex2bn(&rsa->e, keybufB);

Bob can then use rsa* to publicly encrypt the symmetric cypher key which can then be sent to Alice. Alice can then decrypt with the private key

Ben.