Java SSL: client handshake failure and server no cipher suites in common

Question

I am trying to connect from client to server by SSL Socket in Java. Receiving handshake_failure on client side and no cipher suites in common on server side(www).

Server is configured for SSL Tomcat 9, here is connector:

        <Connector SSLEnabled="true" acceptCount="100" clientAuth="false"
            disableUploadTimeout="true" enableLookups="false" maxThreads="25"
            port="8443" keystoreFile="C:/Users/xxx/.keystore" keystorePass="xxx"
            keystoreType="PKCS12"
            protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https"
            secure="true" sslProtocol="TLS" />

Client is Java project. Both sides have certs in keytool. Cert has been exported from keytool and imported at client side by keytool. So, on client side and server side we have imported server.cer. Tomcat starts on port 8443, sockets use port 443.

What I have try? -setting protocols in code -setting cipher suites -changing ports -starting tomcat from 8080 and 8443 port

Server:

    void mySSLServerSocket(int port) throws IOException {

        SSLServerSocketFactory sslServerSocketFactory = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
        SSLServerSocket sslServerSocket = (SSLServerSocket) sslServerSocketFactory.createServerSocket(port);
        sslServerSocket.setEnabledProtocols(new String[] {"TLSv1.2"});
        sslServerSocket.setEnabledCipherSuites(new String[] {"TLS_RSA_WITH_AES_256_CBC_SHA"});
        SSLSocket socket = (SSLSocket) sslServerSocket.accept();
        OutputStream out = socket.getOutputStream();
        InputStream in = socket.getInputStream();
        while(true) {
            int orderNumber = in.read();
            System.out.println("order number readed: " + orderNumber);
        }
    }

Client:

    void mySocket(String ip, int port) throws IOException {

        SSLSocketFactory sslSocketFactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
        SSLSocket sslSocket = (SSLSocket) sslSocketFactory.createSocket(ip, port);
        sslSocket.setEnabledProtocols(new String[] {"TLSv1.2"});
        sslSocket.setEnabledCipherSuites( new String[] {"TLS_RSA_WITH_AES_256_CBC_SHA"});
        sslSocket.startHandshake();
        InputStream in = sslSocket.getInputStream();
        OutputStream out = sslSocket.getOutputStream();
        out.write(1);
        while (in.available() > 0) {
            System.out.print(in.read());
        }

    }

Client error:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at com.company.SocketUtil.mySocket(SocketUtil.java:31)
    at com.company.Main.main(Main.java:15)

Server(error is from web - JSP page):

Type Exception Report

Message no cipher suites in common

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.net.ssl.SSLHandshakeException: no cipher suites in common
    java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
    java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
    java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:254)
    java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.chooseCipherSuite(ServerHello.java:460)
    java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.produce(ServerHello.java:295)
    java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
    java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1102)
    java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:854)
    java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:813)
    java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
    java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
    java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
    java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1180)
    java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1091)
    java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:721)
    java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:804)
    java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:763)
    ServerSocketUtils.mySSLServerSocket(ServerSocketUtils.java:38)
    MusicControl.doPost(MusicControl.java:24)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
Note The full stack trace of the root cause is available in the server logs.

Error says that there is no cipher suites, but I checked supported ciphers and choose one, available for both side. Declared TSL protocol and still no success.

I would like to connect from client to serversuccesfull, but have no idea what can I do next, due no advice found in internet.

I had read something about private key required to do connection. Where is it? I assume, that server.cer have only public key.

Is it proper to use port 443 for sockets, when tomcat runs on 8443? Maybe I should specify those ports somewhere?

Answer 1

One part of your question is about "certs", which - I assume - stands for a certificate.

You are right, a certificate only contains a public key, which is signed by a certification authority (CA). The public key is only the public part of the assymetric key pair (eg RSA). The server needs to have the private key. Otherwise it cannot decrypt content which was encrypted with the public key, nor it can sign content sent to the client.

This said, you need to create a RSA key pair, and make it available to tomcat. Either you let it sign by a well known CA (makes a certificate), or you produce a self singned certificate by using key tool. In the first case there is no special action needed on client side, because java (or browsers) know already the well known CA's public keys which are needed to verify the certificate which is delivered by your server. In case of a self signed certificate you have to import it at client side as a trusted one.