Intermittent SSL error: unable to find valid certification path to requested target

Question

I am working on HttpClient calling rest service over SSL - I am using eclipse as IDE.
I downloaded the rest service certificate using the browser and imported it to my Java keystore under java\jdk1.8.0_144\jre\lib\security\cacerts
Most of the time ecerything is working ok, I am able to get a response,
But I have this Intermittent SSL error: unable to find valid certification path to requested target

I have some theory why is it happening, but I am not sure if the theory is correct and I some questions
about it.

1. is this a possible scenario? : If there are few nodes on the target (under a load balancer) and I only have certificate to one of the nodes?
2. Intermittent permissions issue with eclipse unable to read the keystore?
3. is it possible that the target is using multiple SSL certificates and I only have one?
4. eclipse is using this path java\jdk1.8.0_144\
   I added the certificate to java\jdk1.8.0_144\jre\lib\security\cacerts
   do i need to add it under \java\jdk1.8.0_144\lib\security as well?

If you have any input I will be glad to get any Input.

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 51 more

Answer 1

1. is this a possible scenario? : If there are few nodes on the target (under a load balancer) and I only have certificate to one of the nodes?

If the load-balancer is at the TCP or IP level, maybe. It could certainly allow the different backends to do their own TLS termination, and while all backends for a given domainname (or more generally hostname) should use the same cert(s), somebody could have made a mistake. If the load-balancer is at the HTTP (application) level, it needs to do the TLS termination in order to get the HTTP request (including cookies etc) to process, so only the cert on the load-balancer matters.

This can also occur without an explicit load-balancer. Some systems do informal IP-level balancing by having DNS resolve their name to different addresses at different times -- often, but not necessarily, 'round-robin' through a fixed list. As a result even though you are sending to the same name, your connections actually go to different, varying servers. Others -- especially CDNs like Cloudflare nowadays -- use 'anycast', where the name resolves to a consistent address, but that address can actually go to different physical servers.

See more below.

2. Intermittent permissions issue with eclipse unable to read the keystore?

I doubt it. First, to be clear, when you run a Java program 'in' eclipse, eclipse actually creates a separate JVM to run the specified program, optionally with debugging, and with standard I/O configurable but defaulting to the eclipse Console window. It is that JVM that needs to read the truststore , which by default is the file cacerts. (Java uses keystore files -- and KeyStore class -- to store several kinds of key information, of which certs are only one. The cacerts file is in jks format, but contains only trusted certs of other parties -- nominall CAs, hence the name -- and thus is functionally a truststore.)

If there was a problem reading cacerts you would get a different exception than this.

3. is it possible that the target is using multiple SSL certificates and I only have one?

It's certainly possible. First, if there are actually multiple servers doing termination, as in #1 above, this could easily happen by mistake. If there is actually a single server or device doing termination:

• if you are using multiple hostnames (domainnames) in your URLs which map to the same physical server, it is quite possible that physical server uses different certs for the different 'logical' servers (eg 'VirtualHost' in Apache)

• if you are using different TLS key exchange algorithms for different connections, the server may need to (have and) use different certs. For example if some connections use ECDHE_RSA keyexchange and some use ECDHE_ECDSA, then the server must use a cert containing an RSA key for the first and one containing an ECC key for the second, and those must be different. But unless your code specifies TLS details you normally shouldn't, or you are setting security properties to disable some algorithms only for some executions, Java (JSSE) client will always offer the same ciphersuites (and thus keyexchanges) and a server consistently receiving this same list of options will usually consistently make the same choice. (FYI: for TLS1.3, keyexchange is no longer controlled by ciphersuite, but 1.3 is implemented only in j11 up, not j8.)

• finally, the protocol allows a server to have multiple, different certs for the same hostname(s) and keyexchange(s) and just arbitrarily (even randomly) choose between them. But there is no sensible reason to do so, and I've never seen a server that does.

4. I added the certificate to java\jdk1.8.0_144\jre\lib\security\cacerts do i need to add it under \java\jdk1.8.0_144\lib\security as well?

No. JRE/lib/security/caerts is the right place. As I said above your code is actually running in a JVM, which uses only the JRE, not the JDK in versions where those were different as was the case in j8. (Newer 'modular' Java versions no longer have a fixed JRE defined; the idea is you can create tailored JVMs instead.)

How often is intermittent? If this occurs say once an hour and there are only a few hundred requests per hour, I would turn on javax.net.debug=ssl,handshake,trustmanager and redirect or capture stdout, and look at it to see what cert(s) is/are actually used when and with what results. OTOH if this occurs once per billion requests going through that logfile is much harder.