SSL handshake_failure during ClientHello

Question

I have a Java 8 HTTPS client running on OSX 10.12.3 that is failing early on in the handshake process. Here is the error I'm seeing with SSL Debug info enabled:

trigger seeding of SecureRandom
done seeding SecureRandom
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1472586657 bytes = { 72, 232, 120, 121, 3, 20, 212, 65, 136, 117, 45, 209, 4, 46, 162, 140, 191, 10, 240, 75, 209, 128, 65, 15, 36, 213, 181, 28 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
***
[write] MD5 and SHA1 hashes:  len = 209
0000: 01 00 00 CD 03 03 58 C6   E4 A1 48 E8 78 79 03 14  ......X...H.xy..
0010: D4 41 88 75 2D D1 04 2E   A2 8C BF 0A F0 4B D1 80  .A.u-........K..
0020: 41 0F 24 D5 B5 1C 00 00   64 C0 24 C0 28 00 3D C0  A.$.....d.$.(.=.
0030: 26 C0 2A 00 6B 00 6A C0   0A C0 14 00 35 C0 05 C0  &.*.k.j.....5...
0040: 0F 00 39 00 38 C0 23 C0   27 00 3C C0 25 C0 29 00  ..9.8.#.'.<.%.).
0050: 67 00 40 C0 09 C0 13 00   2F C0 04 C0 0E 00 33 00  g.@...../.....3.
0060: 32 C0 2C C0 2B C0 30 00   9D C0 2E C0 32 00 9F 00  2.,.+.0.....2...
0070: A3 C0 2F 00 9C C0 2D C0   31 00 9E 00 A2 C0 08 C0  ../...-.1.......
0080: 12 00 0A C0 03 C0 0D 00   16 00 13 00 FF 01 00 00  ................
0090: 40 00 0A 00 16 00 14 00   17 00 18 00 19 00 09 00  @...............
00A0: 0A 00 0B 00 0C 00 0D 00   0E 00 16 00 0B 00 02 01  ................
00B0: 00 00 0D 00 1C 00 1A 06   03 06 01 05 03 05 01 04  ................
00C0: 03 04 01 04 02 03 03 03   01 03 02 02 03 02 01 02  ................
00D0: 02                                                 .
main, WRITE: TLSv1.2 Handshake, length = 209
[Raw write]: length = 214
0000: 16 03 03 00 D1 01 00 00   CD 03 03 58 C6 E4 A1 48  ...........X...H
0010: E8 78 79 03 14 D4 41 88   75 2D D1 04 2E A2 8C BF  .xy...A.u-......
0020: 0A F0 4B D1 80 41 0F 24   D5 B5 1C 00 00 64 C0 24  ..K..A.$.....d.$
0030: C0 28 00 3D C0 26 C0 2A   00 6B 00 6A C0 0A C0 14  .(.=.&.*.k.j....
0040: 00 35 C0 05 C0 0F 00 39   00 38 C0 23 C0 27 00 3C  .5.....9.8.#.'.<
0050: C0 25 C0 29 00 67 00 40   C0 09 C0 13 00 2F C0 04  .%.).g.@...../..
0060: C0 0E 00 33 00 32 C0 2C   C0 2B C0 30 00 9D C0 2E  ...3.2.,.+.0....
0070: C0 32 00 9F 00 A3 C0 2F   00 9C C0 2D C0 31 00 9E  .2...../...-.1..
0080: 00 A2 C0 08 C0 12 00 0A   C0 03 C0 0D 00 16 00 13  ................
0090: 00 FF 01 00 00 40 00 0A   00 16 00 14 00 17 00 18  .....@..........
00A0: 00 19 00 09 00 0A 00 0B   00 0C 00 0D 00 0E 00 16  ................
00B0: 00 0B 00 02 01 00 00 0D   00 1C 00 1A 06 03 06 01  ................
00C0: 05 03 05 01 04 03 04 01   04 02 03 03 03 01 03 02  ................
00D0: 02 03 02 01 02 02                                  ......
[Raw read]: length = 5
0000: 15 03 03 00 02                                     .....
[Raw read]: length = 2
0000: 02 28                                              .(
main, READ: TLSv1.2 Alert, length = 2
main, RECV TLSv1.2 ALERT:  fatal, handshake_failure
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Here is what I've tried:

Installed the Java 8 Unlimited Strength JCE jars to:

/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security

and also to:

/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security

I've tested using openssl on the same client machine:

openssl s_client -connect myserver.com:443 -servername myserver.com

The output of that shows:

CONNECTED(00000003)
depth=4 C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
verify return:1
depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = www.myserver.com
verify return:1
---
Certificate chain
 0 s:/CN=www.myserver.com
   i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
   i:/C=US/O=Amazon/CN=Amazon Root CA 1
 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1
   i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
   i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----

encoded cert here

-----END CERTIFICATE-----
subject=/CN=www.myserver.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5963 bytes and written 463 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 0210563871ED6F1FF07FBBBF76D8EA0948E9B38327470DCD4BFDEED8EC44494C
    Session-ID-ctx:
    Master-Key: A806D3B3B6C87B014EAC4CAEDE5093F36F3F2B6E63A367458D4F1897C25BC5A96B301764628901DE431A2EFE96E691F3
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 10800 (seconds)
    TLS session ticket:
    0000 - 32 22 cc 42 2d 93 b9 02-e7 ae 4d f1 a9 db 6f a7   2".B-.....M...o.
    0010 - b1 c6 3a 1b 82 d4 78 ff-ed 69 cf ec 45 1c 67 de   ..:...x..i..E.g.
    0020 - 9d aa 78 17 79 94 9d 39-2a a5 56 71 70 65 3e 06   ..x.y..9*.Vqpe>.
    0030 - b7 65 bd d5 25 4c 61 8d-88 4a d5 4f dd 49 2f ec   .e..%La..J.O.I/.
    0040 - 3b 92 f8 2e b1 f9 87 64-3c be 53 34 88 a0 a4 69   ;......d<.S4...i
    0050 - 79 6b 96 a2 92 22 d9 1e-b8 2f 6f cf 14 a0 47 f6   yk...".../o...G.
    0060 - 29 c8 63 5c d3 95 2c 3a-08 5e d8 72 81 71 0d 96   ).c\..,:.^.r.q..
    0070 - 2f 39 87 4c c5 2c 46 94-5a fd a1 f7 27 e2 56 89   /9.L.,F.Z...'.V.
    0080 - 00 f8 b7 3d 3b 53 22 68-c4 db 2b 41 67 a3 dc 17   ...=;S"h..+Ag...
    0090 - 9e 12 61 74 d7 a8 09 00-d0 af 9f 13 8d 70 df b8   ..at.........p..
    00a0 - 88 7b 3b e6 dc 0a 89 1a-09 2e f9 4d 8b 43 27 52   .{;........M.C'R

    Start Time: 1489430270
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

From what I've researched this is indicating that openssl was able to connect to my server using TLSv1.2 and the ECDHE-RSA-AES128-GCM-SHA256 cipher. I've checked my Java clients list of ciphers in the SSL debug output and I believe that TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 would be the matching cipher and it appears to be enabled, so I would expect the server to negotiate a connection using TLSv1.2 and the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 cipher with my Java client, but that doesn't appear to be the case. I would at least expect it to get to the ServerHello part of the handshake.

Some other things I've tried are importing untrusted certificates from my domain into my global truststore here:

/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts

But if I understand the handshake process correctly the handshake is failing before even getting to the part where it needs to access the truststore. I obtained these certificates by exporting them from my OSX Keychain Access app. FWIW, I have no problems accessing my server from a browser.

Can anybody offer clues as to what is going wrong here. I'm pretty much out of ideas at this point.

Answer 1

Your openssl code explicitly uses -servername to use SNI. Looking at the ClientHello from the Java client it does use SNI. There are several server setups which fail of no SNI extension is present or if it does not match any configured server. My guess is that this happens in your case.