Encryption and decryption error 0x0407106B using OpenSSL

Question

I\'m writing a routine in C that reads a base64 string with the public key and proceeds to encrypt a string. I also test the same string\'s decryption but I\'m getting error

Answer 1

The error you're getting is block type is not 02.

Although Omri is correct that you're passing the wrong data, and you are only going to encrypt 1 byte, the error is because the sizeof( encrypted ) is way too large (2560). In other words your data receiver for RSA_public_encrypt must be a regular unsigned char* pointer, not an unsigned char[2560].

Where you have

unsigned char encrypted[2560] = { 0 }; //X 2560?? RSA_public_encrypt fails.

You should be using

unsigned char *encrypted = (unsigned char*)malloc( rsa_length ) ;
RSA_public_encrypt( DATALEN, (const unsigned char*)str, encrypted, pubKey, PADDING ) ;

Notice the error Omri pointed out, that you used PADDING as the first arg to RSA_public_encrypt, while it should be the DATALEN data length.

If you fix that you'll get a similar error later with the private key decrypt. Fix it and you're on your way.

Answer 2

The problem is that you're trying to decrypt the base64 encoded result. You should try to decrypt the result of the encryption.

That is, instead of:

int resultDecrypt = RSA_private_decrypt( RSA_size(privKey), retencrypted, decrypted, privKey, PADDING);

You should call:

int resultDecrypt = RSA_private_decrypt( RSA_size(privKey), encrypted, decrypted, privKey, PADDING);

Also, there is a problem in the encryption call:

int resultEncrypt = RSA_public_encrypt(PADDING, str, encrypted, pubKey, PADDING);

Why are you passing PADDING as flen? This should be the length of the string (i.e. 4 or 5, depending on whether you want to encrypt the null character).

If you want to write the encrypted string as ASCII (encoded using base64), that's fine. But you have to decode it back before you decrypt it.