Does OpenSSL -sign for ECDSA apply ASN1 encoding to the hash before signing?

Question

This SO question ECDSA sign using OpenSSL without ASN1 encoding the hash states the OpenSSL perfoms ASN1 encoding to the hash before signing it.

In other words it states that OpenSSL performs the following steps when for an Elliptic curve key -sign is called: a. Calculate H = Hash(M) b. Encode H into ASN1 standard- H’ c. Sign H’

And thus to avoid applyting step b it's neccessary to first calculate the digest, and then sign the digest using raw signing - pkeyutl for elliptic curver keys

However when I run BOTH -sign and -dgst+ -pkeyutl I am able to verify the signature using -verify in both cases. This implies that ASN1 encoding is NOT being applied to the hash.

Can anyone throw some light on this topic? I was not able to find documentation in the OpenSSL documentation.

Answer 1

OpenSSL applies ASN.1 DER encoding to the output of the signature.

https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm#Signature_generation_algorithm

1. Calculate e = HASH (m), where HASH is a cryptographic hash function, such as SHA-2.
2. Let z be the Ln leftmost bits of e , where Ln is the bit length of the group order n.
3. Select a cryptographically secure random integer k from [1, n - 1].
4. Calculate the curve point (x1, y1) = k x G.
5. Calculate r = x1 mod n. If r = 0, go back to step 3.
6. Calculate s = (k − 1) x (z + r * dA) mod n. If s = 0, go back to step 3.
7. The signature is the pair (r, s).

The problem is that the ECDSA algorithm ends with math, not with bytes. Two different conventions have arisen for how to turn that pair of numbers into bytes. (Contrast with RSA, whose last step says how to turn the mathematical value back into bytes and declares that byte sequence to be the signature).

Let's assert that we produced a signature (against secp256r1) with r = 67432751043532511959904657272700966685609390316545000351652696368910338707793 and s = 15800012655857962601029927988066555130680701005265153794330961.

ASN.1 DER

Used by X.509/PKIX and OpenSSL. First declared (that I can find, anyways) by RFC 3279, sec 2.2.3:

When signing, the ECDSA algorithm generates two values. These values are commonly referred to as r and s. To easily transfer these two values as one signature, they MUST be ASN.1 encoded using the following ASN.1 structure:

Ecdsa-Sig-Value  ::=  SEQUENCE  {
     r     INTEGER,
     s     INTEGER  }

DER works best if we build it inside out, so let's encode r.

r is an integer, 67432751043532511959904657272700966685609390316545000351652696368910338707793 in decimal, or 951595A548D156D51655159654ADA548D156D5165195159654ADA54D156D5151 hex. ITU-T-REC-X.690-201508 says that the integer gets encoded as a signed big endian value. Since the most significant byte (0x95) has the high bit set this is a negative number, so we need to insert an extra 0x00 to keep the number positive. So r takes 32+1 = 33 bytes:

02 21 (INTEGER, 33 bytes)
   00 (padding byte)
   95 15 95 A5 48 D1 56 D5 16 55 15 96 54 AD A5 48
   D1 56 D5 16 51 95 15 96 54 AD A5 4D 15 6D 51 51

s has decimal value 15800012655857962601029927988066555130680701005265153794330961, or hexadecimal value 9D51655159654ADA548D156D5165195159654ADA54D156D5151. While it starts with hex 9 it's actually 0x09, so no padding byte is required. s only takes 27 bytes of content, because it's so small compared to r.

02 1A (INTEGER, 26 bytes)
   09 D5 16 55 15 96 54 AD A5 48 D1 56 D5 16 51 95
   15 96 54 AD A5 4D 15 6D 51 51

And now we can calculate the size of the containing SEQUENCE to be 63 bytes:

30 3F (CONSTRUCTED SEQUENCE, 64 bytes)
   02 21 (INTEGER, 33 bytes)
      00 (padding byte)
      95 15 95 A5 48 D1 56 D5 16 55 15 96 54 AD A5 48
      D1 56 D5 16 51 95 15 96 54 AD A5 4D 15 6D 51 51
   02 1A (INTEGER, 26 bytes)
      09 D5 16 55 15 96 54 AD A5 48 D1 56 D5 16 51 95
      15 96 54 AD A5 4D 15 6D 51 51

Or, linearized:

30 3F 02 21 00 95 15 95 A5 48 D1 56 D5 16 55 15 
96 54 AD A5 48 D1 56 D5 16 51 95 15 96 54 AD A5 
4D 15 6D 51 51 02 1A 09 51 D5 16 55 15 96 54 AD 
A5 48 D1 56 D5 16 51 95 15 96 54 AD A5 4D 15 6D 
51 51 

IEEE P1363

Used by Windows.

In this format r and s are taken to be big integers of the same byte size as n, then concatenated. Since r uses all 32 bytes it's good to go. s only uses 26 bytes, so it needs 6 leading 0x00 bytes.

// r
95 15 95 A5 48 D1 56 D5 16 55 15 96 54 AD A5 48
D1 56 D5 16 51 95 15 96 54 AD A5 4D 15 6D 51 51
// s
00 00 00 00 00 00 09 D5 16 55 15 96 54 AD A5 48
D1 56 D5 16 51 95 15 96 54 AD A5 4D 15 6D 51 51

Conclusion

So, OpenSSL applies an ASN.1 encoding to the signature, not the hash. The ASN.1 encoding is "more common" (in that it's what's used in ECC certificates). The Windows/IEEE way is easier. The ASN.1 way usually ends up about 6 bytes larger (or 7 bytes larger on average for secp521r1); but could (1 in 2^32 chance) end up the same size, or (1 in 2^40 chance) smaller.

Also, if you ever find yourself on a committee designing a new signature scheme, remember to add the step declaring the wire representation.