Encryption and decryption error 0x0407106B using OpenSSL

Question

I\'m writing a routine in C that reads a base64 string with the public key and proceeds to encrypt a string. I also test the same string\'s decryption but I\'m getting error

Answer 1

The error you're getting is block type is not 02.

Although Omri is correct that you're passing the wrong data, and you are only going to encrypt 1 byte, the error is because the sizeof( encrypted ) is way too large (2560). In other words your data receiver for RSA_public_encrypt must be a regular unsigned char* pointer, not an unsigned char[2560].

Where you have

unsigned char encrypted[2560] = { 0 }; //X 2560?? RSA_public_encrypt fails.

You should be using

unsigned char *encrypted = (unsigned char*)malloc( rsa_length ) ;
RSA_public_encrypt( DATALEN, (const unsigned char*)str, encrypted, pubKey, PADDING ) ;

Notice the error Omri pointed out, that you used PADDING as the first arg to RSA_public_encrypt, while it should be the DATALEN data length.

If you fix that you'll get a similar error later with the private key decrypt. Fix it and you're on your way.