Compute a signature with private key in Server Key Exchange

Question

On https://tls.ulfheim.net/ there is an example showing how to compute a signature in the section of " Server Key Exchange ".

https://i.ibb.co/Y7fbkDw/1.jpg (This image shows the Server Key Exchange section on the website that I refer to. Could not embed the image in this post.)

Whatever I try I dont get the same output as the one on that website, I dont understand why.

I've tried storing the same data in two different ways, then use the same openssl command that they use on their example. None of the methods gave the same output.

Method 1.

char hex[] = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, 0x20, 0x9f, 0xd7, 0xad, 0x6d, 0xcf, 0xf4, 0x29, 0x8d, 0xd3, 0xf9, 0x6d, 0x5b, 0x1b, 0x2a, 0xf9, 0x10, 0xa0, 0x53, 0x5b, 0x14, 0x88, 0xd7, 0xf8, 0xfa, 0xbb, 0x34, 0x9a, 0x98, 0x28, 0x80, 0xb6, 0x15 };
ofstream myfile("c:/hex1.txt", ios::binary);
myfile.write(hex, sizeof hex);

then:

openssl dgst -hex -sign server.key -sha256 hex1.txt

Method 2.

I had this data stored in hex2.txt (as ASCII):

\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x20\x9f\xd7\xad\x6d\xcf\xf4\x29\x8d\xd3\xf9\x6d\x5b\x1b\x2a\xf9\x10\xa0\x53\x5b\x14\x88\xd7\xf8\xfa\xbb\x34\x9a\x98\x28\x80\xb6\x15

then:

openssl dgst -hex -sign server.key -sha256 hex2.txt

Answer 1

Method 1

You left out curve_info . The applicable RFC 4492 section 5.4 updated for TLS1.2 by RFC 5246 appendix A.7 actually defines the signature to be over in effect client_random + server_random + SKX_params where SKX_params is type ServerECDHParams and consists of ECParameters curve_params and ECPoint public -- these are what ulfheim labels Curve Info and Public Key.

With the correct data I get the correct result:

$ od -tx1 70148855.bin
0000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000020 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
0000040 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
0000060 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
0000100 03 00 1d 20 9f d7 ad 6d cf f4 29 8d d3 f9 6d 5b
0000120 1b 2a f9 10 a0 53 5b 14 88 d7 f8 fa bb 34 9a 98
0000140 28 80 b6 15
$ openssl sha256 <70148855.bin -sign $privkey -hex
(stdin)= 0402b661f7c191ee59be45376639bdc3d4bb81e115ca73c8348b525b0d2338aa144667ed9431021412cd9b844cba29934aaacce873414ec11cb02e272d0ad81f767d33076721f13bf36020cf0b1fd0ecb078de1128beba0949ebece1a1f96e209dc36e4fffd36b673a7ddc1597ad4408e485c4adb2c873841249372523809e4312d0c7b3522ef983cac1e03935ff13a8e96ba681a62e40d3e70a7ff35866d3d9993f9e26a634c81b4e71380fcdd6f4e835f75a6409c7dc2c07410e6f87858c7b94c01c2e32f291769eacca71643b8b98a963df0a329bea4ed6397e8cd01a110ab361ac5bad1ccd840a6c8a6eaa001a9d7d87dc3318643571226c4dd2c2ac41fb

Method 2

is completely incorrect. \x00 etc is a notation used in source code for C and C++ (as you did in Method 1) and a few other languages like Java, JS/ES, and Python, as well as certain tools like the printf command in Unix and the awk program. But it does not work other places, and in particular does not work in files read by OpenSSL (at least for data; it might work in the config file, I'd have to check). Your method two (hashes and) signs the bytes 0x5c 0x78 0x30 0x30 0x5c 0x78 0x30 0x31 which represent the characters \ x 0 0 \ x 0 1 etc, not the bytes 0x00 0x01 etc, and unsurprisingly this is completely different and wrong.