Using the OpenSSL API, I have extracted a custom extension from a X.509v3 certificate with:
X509_EXTENSION* ex = X509_get
@Francois pointed me to the ASN1_get_object() function. That function is appropriate for this scenario where the certificate extension contains only a single value.
ASN1_get_object() takes a pointer to a pointer to a C buffer that contains a DER encoded object. It returns the data itself (by adjusting the pointer), the length of the data, the ASN.1 tag value and the ASN.1 object class.
ASN1_OCTET_STRING* octet_str = X509_EXTENSION_get_data(extension);
const unsigned char* octet_str_data = octet_str->data;
long xlen;
int tag, xclass;
int ret = ASN1_get_object(&octet_str_data, &xlen, &tag, &xclass, octet_str->length);
printf("value: %s\n", octet_str_data);