RSA encrypt then decrypt fails with “oaep decoding error”

问题

I am trying to debug a simple openssl program but cannot quite get what openssl means with "oaep decoding error". I have reduced my program to this, test, one:

#include <iostream>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <assert.h>
#include <memory>
#include <list>
#include <algorithm>
#include <string.h>

static const std::string private_key =
R"(-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAriDoH3gBbJo+SojeL5j+4yQumXgnjhrt5+FChBxOfvTcyczz
p5qlUNqLzQQcQ/a+XR5qUhaA4l97DgNseFNyoYHIxrB5t+BQw27Q+UuUYYaIwJqZ
6r2PVCnQF9WPqqWdzBN6+13IlreH2XX4qy47kuLI3lcPlP5qhYaDogpZCl7lN7Oe
Qj2FAPZ1nkV+PL4RYfWZbBymUz0C105ytT7PWgLAZVvag8kHiNXRYv4Ksc9SJ3AE
ZriZ1tzpJ6/ZkI3vfJZqCeclELQ8zGxb8CbfSd4mHoHCVY7t1h+BNM4zUxSjN8d7
bctYJwnfBtztRFN2uTotDJs6njsoxfmh4/SzAQIDAQABAoIBAAGTWKZYPqMN7jxc
aq5BkyTZAfbviGQXyElN1308iFVLv+evjBDbLF3D7HnpbJwM0oIjMVEW1Qm3VXS2
AThBgQsHEpsBo8hPJkvuZ8OptGkBf6FGhNgD6RUY38Inc4pWv0vGbVly6sq6VGda
Uuqxm2Zj2O9yGDj/6FTW97/ymgWm/FfKczg/zGtjdog67W8LvvtmAj5ynSuimOP8
mOINPjewIbcl7rKvxcMNrOXKsRWwVxTNXdMNMsXd1Figw022KTqdiazQ/DPIXU6M
f8H+U/gS5QZRIAF8i0r3cvq6ai26dX0OFtsoizqG4qlRNwtQ+wyRsilZKiKnFuMY
bt1pRBUCgYEA1TlAT/Ui4TBdgGmm0Rlj7JKJENnpDKIFE8bP6Vy8SwBmp5MiRofE
TMne4BBKLcFcslCJrFvjl7+v4B9a2de7hJYqtevrXjM91vwFhc6z0m27vv6MKStQ
3uKX8+0RGHQ3j53kAvLxFSuAqYQ+gf9IAuyG0gpMABRvj0/8HY3T7tMCgYEA0Q/O
0og9UbXh8y3yI94ztczWdIQERyEhQiGNRUnHCqO2QbZQ9Nm190Jx/8yew03xpPVb
fyWWfKqO8Kjg5np0w37porI0UmfLZ5QMC+GFMq0jOUXidsvkyoWOe4D8LII0L98k
sjihHBlGNrfFjEgOUQaoreB+8F07m/iofRCROlsCgYAPUUGRfOa4jqTo6K4XL1/C
SvSVxVG8mpcKyKl+9i6ApNK7DxLTRkWPzqC4L/NkPhPOq4J4Y1GCQT79NsNsCtdp
uu/uibgq2DuFCi3LYwIAB+oI2nhvLLFukZCg8VLdEtw68PjETXeMMcfYZaun4xLl
QuCcjijPiKhK/0/5P4sOCQKBgHsi7XXRqxRapdg/ArUfpqN5IAOG0qI2oEk8S+I4
v1TD8pCn2u0s4mHdsBmzovt0CFVZ8udj80xAhWq4facjD20qbmBWyDyVSBgc+i9x
SKv9kJamU+oW1A55NeAGrAFnO2fK7elPM43CUTnfairjMhOFcYrghMP8liSbBFqN
jIyrAoGAVGZQVZgicmSppbBuZYJPOuegPVYhncq3XtBZCtEGGlMtQpKgz+GRhyvT
Ar/HC7xnS5Gjfyjj6eGHsBAjTsE4t38qD4nxQXzBmAQQ1/7/iq3WNu63OV2q4GRh
wChOO0pcJPOZfWtvKiy7hbN09e0nt5blX1yqe6LdO7mACWli/Ss=
-----END RSA PRIVATE KEY-----)";

static const std::string public_key = 
R"(-----BEGIN RSA PUBLIC KEY-----
MIIBCAKCAQEAx5WRSyfFVe/JbPYnswghuMj5Nzo9YG82Z7ehyI/dbjkcdCIzTlKd
QcMvSUZafAnM9p3xnBrgbKaNltaVNrZNyN6A2ou0PQgms7ykJ67G9Hbbs/uo0rPS
GS4pYw0QiOvoYXjGqbOzQjXbAV7ez05XRb43nRdZUFO0LLvEp2VfaTL7WWzaan6r
Ve6p8t8JIwpWSn7njBYH2XPNJj1NccpvD+kT1kGn6kWZfmFBzR7Bw2+rW+rpt02F
4arxXfvzDYhZdxLKb7m2KqwZTiug2HoD5AY9l3GzRIdNvXIDP87XTl4960lpg8cI
8XuTSLFjSx0fvlXXFwTcgMLv7Q0+ISSXQwIBJQ==
-----END RSA PUBLIC KEY-----)";

static inline 
void check_errors()
{
    const int error = ERR_get_error();
    if ( error != 0 )
    {
        std::cerr << ERR_reason_error_string( error ) << '\n';
        assert(0);
    }
}

int main()
{
    ERR_load_crypto_strings();

    // load rsa keys into a memory buffer
    BIO * bio_private = BIO_new_mem_buf( (char*) private_key.data(), private_key.size()  ); check_errors();
    BIO * bio_public  = BIO_new_mem_buf( (char*) public_key.data(),  public_key.size()  ); check_errors();

    // use the memory buffer to create a RSA struct
    RSA * rsa_private = RSA_new(); check_errors();
    RSA * rsa_public  = RSA_new(); check_errors();

    PEM_read_bio_RSAPrivateKey( bio_private, &rsa_private, nullptr, 0 ); check_errors();
    PEM_read_bio_RSAPublicKey(  bio_public,  &rsa_public,  nullptr, 0 ); check_errors();

    // encrypt this text ...
    std::string text_to_encrypt = "hello";

    // ... in this buffer
    std::vector< unsigned char > buffer;
    buffer.resize( RSA_size( rsa_private ) );

    const int flen = RSA_public_encrypt( 
                text_to_encrypt.size(), // shorter than the key in my test
                (const unsigned char*)text_to_encrypt.data(),
                &buffer[0],
                rsa_public,
                RSA_PKCS1_OAEP_PADDING
            ); check_errors();

    // now decrypt the buffer back to the original text
    std::vector< unsigned char > buffer_decrypt;
    buffer_decrypt.resize( RSA_size( rsa_private ), '\0' );

    RSA_private_decrypt(
                flen, 
                &buffer[0],
                &buffer_decrypt[0],
                rsa_private,
                RSA_PKCS1_OAEP_PADDING
            ); check_errors();

    // and print it
    std::cout << (char*) &buffer_decrypt[0] << '\n';
}

The check_errors() located just after RSA_private_decrypt triggers with the error mentioned and I have no idea why.

EDIT: slightly improved the code by using the value returned by RSA_public_encrypt. In practice it does not affect the results.

回答1:

The check_errors() located just after RSA_private_decrypt triggers with the error mentioned and I have no idea why.

Your public key does not match the private key.

When I add the following after the keys are read:

FILE* ff = fopen("temp.pem", "w");
PEM_write_RSAPublicKey(ff, rsa_private);

And then cat the public key:

$ cat temp.pem 
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEAriDoH3gBbJo+SojeL5j+4yQumXgnjhrt5+FChBxOfvTcyczzp5ql
UNqLzQQcQ/a+XR5qUhaA4l97DgNseFNyoYHIxrB5t+BQw27Q+UuUYYaIwJqZ6r2P
VCnQF9WPqqWdzBN6+13IlreH2XX4qy47kuLI3lcPlP5qhYaDogpZCl7lN7OeQj2F
APZ1nkV+PL4RYfWZbBymUz0C105ytT7PWgLAZVvag8kHiNXRYv4Ksc9SJ3AEZriZ
1tzpJ6/ZkI3vfJZqCeclELQ8zGxb8CbfSd4mHoHCVY7t1h+BNM4zUxSjN8d7bctY
JwnfBtztRFN2uTotDJs6njsoxfmh4/SzAQIDAQAB
-----END RSA PUBLIC KEY-----

Bingo! It does not match the hard coded public key.

If you want to get beyond it for testing, just use rsa_private for encrypting. Only the public portion of the key is used during RSA_public_encrypt:

int flen = RSA_public_encrypt( 
               text_to_encrypt.size() + 1,
               text_to_encrypt.data(),
               &buffer[0],
               rsa_private,
               RSA_PKCS1_OAEP_PADDING
            );

Or, just update the public key now that you know what it should be.

来源：https://stackoverflow.com/questions/25222068/rsa-encrypt-then-decrypt-fails-with-oaep-decoding-error