Java 1.7 Subject Hash of X.509 Certificate OpenSSL 1.0+ Compatible

Question

I\'ve been struggling with this for a couple of days. I\'m working on a Java 1.7 app running in an embedded Linux environment. OpenSSL is not available and I don\'t have control

Answer 1

This answer was the closest I've found to a good one, but it's far from that
There are a few misconceptions in it
- X509_NAME_hash returns a long unsigned long X509_NAME_hash(X509_NAME *x)
- it's not the 10th character that needs to be 0x0c, rather the 1st character of the value
- converting the text to lower case does not solve the problem

In order to overcome these, I've started using the X500Name instead of the X500Principal as an input parameter a conversion between the two can be done easily
The reason for this is that the X500Name exposes the RDN array from where we can retrieve the values (here, i'm ignoring the multi value options and only using the first)
retrieving the name allows me to do canonical conversion (not only lowercase) and to know where it start, to replace the 1st byte to 0x0c

Updated code now includes the full solution without multiple byte to string and back conversions

public static long calculateX500NameHash(X500Name name) throws IOException, NoSuchAlgorithmException {
    byte[] nameEncoded = name.getEncoded();

    final ASN1Sequence asn1Sequence = (ASN1Sequence) ASN1Primitive.fromByteArray(nameEncoded);
    List rdnList = new ArrayList<>();
    int length = 0;
    for (ASN1Encodable asn1Set : asn1Sequence.toArray()) {
        byte[] bytes = ((ASN1Set) asn1Set).getEncoded();
        length += bytes.length;
        rdnList.add(bytes);
    }
    byte[] nameBytes = new byte[length];
    int counter = 0;
    int addedItems = 0;
    for (RDN rdn : name.getRDNs()) {
        // Get original encoded RDN value
        byte[] encoded = rdn.getFirst().getValue().toASN1Primitive().getEncoded();
        // Get the RDN value as string without the prefix
        StringBuilder content = new StringBuilder();
        for (int j = 2; j < encoded.length; j++) {
            content.append((char) encoded[j]);
        }
        // canonicalize the string
        byte[] updateContent = IETFUtils.canonicalize(content.toString()).getBytes(StandardCharsets.UTF_8);
        // create new byte[] with the updated prefix and canonicalized string
        byte[] updated = new byte[encoded.length];
        updated[0] = 0x0c;
        updated[1] = encoded[1];
        System.arraycopy(updateContent, 0, updated, 2, updateContent.length);
        // get full RDN with type prefix
        byte[] rdnFromList = rdnList.get(counter);
        int fullLength = rdnFromList.length;
        int valueLength = encoded.length;
        // Additional check, expect to always return true
        if (isMatchingTheEnd(rdnFromList, encoded)) {
            int prefixLength = (fullLength - valueLength);
            // add the beginning of the full RDN to the `nameBytes` array without the value
            System.arraycopy(rdnFromList, 0, nameBytes, addedItems, prefixLength);
            // add the updated value to the `nameBytes` array
            System.arraycopy(updated, 0, nameBytes, addedItems + prefixLength, valueLength);
        } else {
            // safeguard
            System.arraycopy(rdnFromList, 0, nameBytes, addedItems, fullLength);
        }
        addedItems += fullLength;
        ++counter;
    }
    return getHashFromByteArray(nameBytes) & 0xffffffffL;
}

private static boolean isMatchingTheEnd(byte[] fullRdn, byte[] rdnValue) {
    int fullRdnLength = fullRdn.length;
    int rdnValueLength = rdnValue.length;
    if (fullRdnLength > rdnValueLength) {
        int prefixLength = fullRdnLength - rdnValueLength;
        for (int i = 0; i < rdnValueLength; i++) {
            if (fullRdn[prefixLength + i] != rdnValue[i]) {
                return false;
            }
        }
        return true;
    }
    return false;
}


private static long getHashFromByteArray(byte[] nameBytes) throws NoSuchAlgorithmException {
    byte[] digest = MessageDigest.getInstance("SHA1").digest(nameBytes);
    return (((digest[0] & 0xff))
            | (((digest[1] & 0xff) << 8))
            | (((digest[2] & 0xff) << 16))
            | (((digest[3] & 0xff) << 24)));
}

Hope this helps someone