TLS1.2是否使用配置了SunPKCS11-NSS提供程序的JSSE在FIPS模式下与NSS一起使用

Question

We are trying to get TLS1.2 working in FIPS mode in the following environment. When attempting to write to an SSLSocket created with a SSLContext for protocol “TLSv1.2” it fails with the error java.security.NoSuchAlgorithmException: no such algorithm: SunTls12RsaPremasterSecret for provider SunPKCS11-NSS .

The environment:

• Java: OpenJDK 1.8.0_25-b17 (client and server)
• OS: CentOS release 6.5 (Final)
• NSS: nss-3.16.2.3-3.el6_6.x86_64
• FIPS mode is enabled on keystore using modutil.

I believe the answer is NO, TLS1.2 does not work with NSS 3.16+ in FIPS mode. I need to determine decisively:

• whether this is expected to work and I should therefore continue digging to find out what we are doing wrong (in which case pointers and ideas to make this work are requested) or
• it is not supported and I should therefore expect it to be supported in which future NSS version

Here is the evidence collected so far:

1. The error that occurs when trying to use TLS1.2 is java.security.NoSuchAlgorithmException: no such algorithm: SunTls12RsaPremasterSecret for provider SunPKCS11-NSS. An identical error message is disussed in this post , with indications that it means TLS 1.2 is not supported.
2. FIPS validation of NSS was last performed in 2010 on NSS version 3.12.14. TLS1.2 support is not mentioned in any NSS release notes until 3.15.1. A related question is posted here , but there is no specific mention of TLSv1.2.
3. CentOS 6.5 release notes indicate that FIPS mode and TLS1.2 are supported, but there is no mention of whether the two are supported at the same time.
4. Our tests using TLS1.2 SSLContext (client and server) work when the system is NOT in FIPS mode. Running the exact same test after running modutil –fips true results in the described error and console output provided below.

The relevant javax.net.debug output follows:

*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1409235389 bytes = { 122, 104, 164, 187, 130, 152, 6, 95, 250, 230, 146, 99, 164, 228, 116, 203, 188, 51, 48, 140, 196, 35, 87, 33, 228, 67, 15, 120 }
Session ID:  {}
Cipher Suites: [TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
***

TRIMMED

%% Negotiating:  [Session-1, TLS_RSA_WITH_AES_128_CBC_SHA256]
*** ServerHello, TLSv1.2
RandomCookie:  GMT: 1409235389 bytes = { 217, 13, 11, 142, 204, 139, 77, 178, 239, 246, 177, 116, 225, 208, 217, 77, 128, 106, 206, 72, 40, 229, 46, 232, 54, 172, 74, 41 }
Session ID:  {84, 255, 58, 189, 125, 207, 159, 166, 144, 6, 19, 157, 173, 113, 80, 88, 204, 69, 101, 231, 227, 242, 144, 59, 174, 147, 158, 65, 14, 78, 182, 52}
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: 
***
Cipher suite:  TLS_RSA_WITH_AES_128_CBC_SHA256

TRIMMED

*** ServerHelloDone
[read] MD5 and SHA1 hashes:  len = 4
0000: 0E 00 00 00                                        ....
Thread-0, handling exception: javax.net.ssl.SSLKeyException: RSA premaster secret error
%% Invalidated:  [Session-1, TLS_RSA_WITH_AES_128_CBC_SHA256]

12:41:01.684 [Thread-0] ERROR c.p.p.c.i.SslContextTestContainer - unexpected Exception
javax.net.ssl.SSLKeyException: RSA premaster secret error
    at sun.security.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:86) ~[na:1.8.0_25]
    at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:880) ~[na:1.8.0_25]
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:344) ~[na:1.8.0_25]
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:936) ~[na:1.8.0_25]
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:871) ~[na:1.8.0_25]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043) ~[na:1.8.0_25]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343) ~[na:1.8.0_25]
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728) ~[na:1.8.0_25]
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_25]
    at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_25]
    at com.polycom.pillars.certificate.internal.SslContextTestContainer$ClientThread.doit(SslContextTestContainer.java:193) ~[bin/:na]
    at com.polycom.pillars.certificate.internal.SslContextTestContainer$SslConnectionThread.run(SslContextTestContainer.java:127) ~[bin/:na]
Caused by: java.security.NoSuchAlgorithmException: no such algorithm: SunTls12RsaPremasterSecret for provider SunPKCS11-NSS
    at sun.security.jca.GetInstance.getService(GetInstance.java:101) ~[na:1.8.0_25]
    at javax.crypto.JceSecurity.getInstance(JceSecurity.java:109) ~[na:1.8.0_25]
    at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:287) ~[na:1.8.0_25]
    at sun.security.ssl.JsseJce.getKeyGenerator(JsseJce.java:274) ~[na:1.8.0_25]
    at sun.security.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:77) ~[na:1.8.0_25]

Answer 1

Indeed the provider does not support TLS1.2. The bug tracking this issue is: https://bugs.openjdk.java.net/browse/JDK-8029661

Answer 2

The exceptions are coming since the server still attempts to negotiate in TLS1.2 even though it not supported by the provider NSS. To avoid this exception and to proceed to use TLS1.1 add the following lines under java.security.

jdk.tls.disabledAlgorithms=SSLv3,TLSv1.2