Bad Padding Exception - RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING in pkcs11

Question

My application is accessing e-Token for decrypting the response coming from the server

The session key from the server is encrypted using :-

RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING

I am using SunPKCS11 Provider for implementing the access to crypto token. Whenever i try to decrypt this using the above mechanishm i.e. with RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING i am getting :-

**javax.crypto.BadPaddingException: doFinal() failed  
    at sun.security.pkcs11.P11RSACipher.implDoFinal(P11RSACipher.java:328)  
    at sun.security.pkcs11.P11RSACipher.engineDoFinal(P11RSACipher.java:353)  
    at javax.crypto.Cipher.doFinal(DashoA13*..)

The following is my code :-

private static final String TRANSFORMATION = "RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING";
private static final String SECURITY_PROVIDER = "BC";
private static final String DIGEST_ALGORITHM = "SHA-256";
private static final String MASKING_FUNCTION = "MGF1";

The code snippet where the error is coming is as follows :-

private byte[] decryptSecretKeyData(byte[] encryptedSecretKey, byte[] iv, PrivateKey privateKey) throws Exception {

        try {
            Cipher rsaCipher = Cipher.getInstance(TRANSFORMATION, SECURITY_PROVIDER);

            System.out.println("Cipher block initialized"); - **Printed**
            PSource pSrc = (new PSource.PSpecified(iv));
            System.out.println("PSource inisitialized"); - **Printed**


            rsaCipher.init(Cipher.DECRYPT_MODE, privateKey,
                    new OAEPParameterSpec(DIGEST_ALGORITHM, MASKING_FUNCTION,
                            MGF1ParameterSpec.SHA256, pSrc));


            System.out.println("Here after cipher initilaization");  - **Not Printed***

            return rsaCipher.doFinal(encryptedSecretKey);
        } catch (GeneralSecurityException e) {
            System.out.println("GeneralSecurityException is "+e.getMessage());
            throw new Exception("Failed to decrypt AES secret key using RSA.", e);
        }
    }

I am Stuck in here and unable to decrypt the OAEP Transformation.

Answer 1

You obtain a non-extractable private key P11Key.P11PrivateKey from the dongle. It cannot be used outside PKCS11 provider, thus, SunPKCS11 provider should be used for operations with that key.

Unfortunately SunPKCS11 provider doesn't support OAEP padding, making it more difficult. Encryption still can be done with BouncyCastle, but decryption can be done with no padding and SunPKCS11 provider. keyLength parameter is RSA key modulus length in bits (1024,2048 etc).

private void testEncryption(byte[] plainText, PrivateKey privateKey, PublicKey publicKey, int keyLength) throws GeneralSecurityException {

    System.out.println("Plain text: " + DatatypeConverter.printHexBinary(plainText));

    Provider bcProvider = new BouncyCastleProvider();
    Cipher rsaCipher = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING", bcProvider);
    rsaCipher.init(Cipher.ENCRYPT_MODE, publicKey);
    byte[] cipherText = rsaCipher.doFinal(plainText);

    System.out.println("Cipher text: " + DatatypeConverter.printHexBinary(cipherText));

    Provider pkcs11provider = new SunPKCS11("C:\\Users\\manishs525\\pkcs11.cfg");
    Cipher rsaCipher2 = Cipher.getInstance("RSA/ECB/NoPadding", pkcs11provider);
    rsaCipher2.init(Cipher.DECRYPT_MODE, privateKey);
    byte[] paddedPlainText = rsaCipher2.doFinal(cipherText);

    /* Ensure leading zeros not stripped */
    if (paddedPlainText.length < keyLength / 8) {
        byte[] tmp = new byte[keyLength / 8];
        System.arraycopy(paddedPlainText, 0, tmp, tmp.length - paddedPlainText.length, paddedPlainText.length);
        System.out.println("Zero padding to " + (keyLength / 8));
        paddedPlainText = tmp;
    }           

    System.out.println("OAEP padded plain text: " + DatatypeConverter.printHexBinary(paddedPlainText));

    OAEPParameterSpec paramSpec = new OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA1,
            PSource.PSpecified.DEFAULT);
    RSAPadding padding = RSAPadding.getInstance(RSAPadding.PAD_OAEP_MGF1, keyLength / 8, new SecureRandom(), paramSpec);
    byte[] plainText2 = padding.unpad(paddedPlainText);

    System.out.println("Unpadded plain text: " + DatatypeConverter.printHexBinary(plainText2));
}

Notes:

• RSA/ECB/NoPadding is not implemented for SunPKCS11 before JDK1.7.
• This example was tested with BouncyCastle 1.50 and JDK 1.7

Answer 2

I have found the issue is that the implementation of SunJCE's Cipher "RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING" is not compatible with other implementations (BouncyCastle/IAIK/PKCS11)

When setting AlgorithmParameters (with OAEPParameterSpec) an exception is thrown (javax.crypto.BadPaddingException)

Refer : Problems with Cipher "RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING" Bug Details