Two-way (mutual) SSL with Glassfish3/4 or Tomcat 8 and self-signed certificates

Question

I'm trying to implement two-way (mutual) SSL authentication, but I constantly get the following exception either on Glassfish3/4 and Tomcat 8 servers (the stacktrace is from Tomcat 8):

10-Feb-2016 17:13:41.579 SEVERE [http-nio2-18443-exec-2] org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun Error running socket processor
java.lang.RuntimeException: Field length overflow, the field length (7189180) should be less than 65536
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1373)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:529)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:807)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:775)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.apache.tomcat.util.net.SecureNio2Channel.handshakeUnwrap(SecureNio2Channel.java:394)
    at org.apache.tomcat.util.net.SecureNio2Channel.handshakeInternal(SecureNio2Channel.java:267)
    at org.apache.tomcat.util.net.SecureNio2Channel.handshake(SecureNio2Channel.java:204)
    at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1064)
    at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.run(Nio2Endpoint.java:1046)
    at org.apache.tomcat.util.net.Nio2Endpoint.processSocket0(Nio2Endpoint.java:598)
    at org.apache.tomcat.util.net.Nio2Endpoint.processSocket(Nio2Endpoint.java:583)
    at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:83)
    at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:76)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker$2.run(Invoker.java:218)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

With long searching, I only found this link which shows the same problem but on JBoss. Is this really related to the size/number of entries in the key/truststore? I now have ~66.000 certificates stored, but as development goes on, we expect way more (like, hundreds of thousands).

With an almost empty keystore (only the server and one test client certificate in it), testing the SSL connection with curl is succesful...

UPDATE

Okay, I think I figured out that it does related to truststore. Here is the code snippet from sun.security.ssl.HandshakeMessage.CertificateRequest :

        // put certificate_authorities
        int len = 0;
        for (int i = 0; i < authorities.length; i++) {
            len += authorities[i].length();
        }

        output.putInt16(len);

The exception is fired in the putInt() method, so I assume I really have too much certificates stored. I'm generating self-signed certificates right now, so I have to have every generated certificate as a CA on the server to trust the clients, and this must be the problem. The question now is that is there a way to generate certificates from Java that are not self-signed (eg I could use glassfish's self-signed certificate as a CA - or can I)?

Answer 1

Okay, I made it to work by creating certificates with issuer set to server's own certificate (in Glassfish, with alias s1as particularly) and signing it with the server's private key. Now I don't have to store anything on the server as it recognises clients because it knows their issuer as CA in its trustStore (essentially, itself).