Connect to websocket with self-signed certificate in java

Question

I need to use Java to connect to a WebSocket server that is using a self-signed certificate. I'm trying to use the Jetty library and am pretty new at Java but I am finding it very difficult to figure out what needs to be done. I can connect using NodeJS very simply:

const WebSocket = require('ws');
const ws = new WebSocket('wss://192.168.100.220:9000/', ['ws-valence'], {
  rejectUnauthorized: false,
});

However, modifying the example I found on the Jetty docs doesn't get me very far.

I implemented a basic client that works well with an echo test server, like in the example linked above. Then I went on to configure it with my own protocol and IP Address:

  private static void connectToBasestation() {

//    String destUri = "ws://echo.websocket.org";
    String basestationUri = "wss://192.168.100.220:9000/";
    SslContextFactory ssl = new SslContextFactory(); // ssl config
    ssl.setTrustAll(true); // trust all certificates
    WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client
    BasestationSocket socket = new BasestationSocket();
    ArrayList<String> protocols = new ArrayList<String>();
    protocols.add("ws-valence");

    try
    {
        client.start();
        URI bsUri = new URI(basestationUri);
        ClientUpgradeRequest request = new ClientUpgradeRequest();
        request.setSubProtocols(protocols);
        client.connect(socket, bsUri, request);
        System.out.printf("Connecting to : %s%n", bsUri);

        // wait for closed socket connection.
        socket.awaitClose(5,TimeUnit.SECONDS);
    }
    catch (Throwable t)
    {
        t.printStackTrace();
    }
    finally
    {
        try
        {
            client.stop();
        }
        catch (Exception e)
        {
            e.printStackTrace();

        }
    }
  }

However, I'm getting an UpgradeException with 0 null as the values and my onConnect method is never getting called. I'm guessing this is a security issue, but I can't be certain since the server is an old machine -- a bit of a black box. But I'm thinking maybe something is wrong with my approach? Can anyone lend any advice here?

Edit 1: Included trustful SSL factory as suggested. It did not change anything, including the stack trace from below.

Edit 3: There is a similar question listed above, but this is different since 1) I'm getting a different error code and 2) Adding a trustful SSL factory does not solve the issue.

Edit 2: Here is the stack trace I am getting from my OnError below:

Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
    at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:681)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:128)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
    at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    ... 3 more
org.eclipse.jetty.websocket.api.UpgradeException: 0 null
    at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:522)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:216)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:208)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:470)
    at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:552)
    at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:156)
    at org.eclipse.jetty.client.HttpSender.terminateRequest(HttpSender.java:381)
    at org.eclipse.jetty.client.HttpSender.abort(HttpSender.java:566)
    at org.eclipse.jetty.client.HttpSender.anyToFailure(HttpSender.java:350)
    at org.eclipse.jetty.client.HttpSender$CommitCallback.failed(HttpSender.java:717)
    at org.eclipse.jetty.client.http.HttpSenderOverHTTP$HeadersCallback.failed(HttpSenderOverHTTP.java:310)
    at org.eclipse.jetty.io.WriteFlusher$PendingState.fail(WriteFlusher.java:263)
    at org.eclipse.jetty.io.WriteFlusher.onFail(WriteFlusher.java:516)
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint$FailWrite.run(SslConnection.java:1251)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

A TLS/SSL handshake error is rather generic.

You don't know what part of the TLS/SSL handshake the issue occurred in.

You can use -Djavax.net.debug=all command line option on Java to see the raw details of the TLS/SSL handshake, and this might be a good place to start troubleshooting your issues with.

Some options ...

---

For certificate name issues

If you connect to a server and the provided certificate does not match the hostname you used in the URI to connect, this is a violation of the endpoint identification algorithm present in Java itself.

Example Scenario:

1. You connect to wss://192.168.1.0:8443/chat
2. The certificate reports itself as chatserver.acme.com

This is a violation, as the hostname in the URI 192.168.1.0 does not match the certificate chatserver.acme.com

This is especially common when testing with wss://localhost or wss://127.0.0.1

You can tell Java to not perform the Endpoint Identification check like this ...

SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setEndpointIdentificationAlgorithm(null); // disable endpoint identification algorithm.
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client

⚠️ WARNING: This is not recommended, and can easily allow for man-in-the-middle attacks!

---

For a certificate trust issues

Try enabling trust for all certificates.

1. Enable SSL/TLS for WebSocket Client
2. Trust All Certificates on the SSL/TLS Configuration

Example (assuming Jetty 9.4.19.v20190610 or newer):

SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setTrustAll(true); // trust all certificates
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client

⚠️ WARNING: This is not recommended, and can easily allow for man-in-the-middle attacks!

---

For certificate algorithm issues

The algorithm used to create the certificate will limit the available Cipher Suites made available during the TLS/SSL handshake.

For example, If the server only had a DSA certificate (known vulnerable), then none of the RSA or ECDSA certificates would be available.

The number of bits used to create the certificate is also relevant, as if the server certificate had too few, then Java itself will reject it.

If you are in control of the server certificate, make sure you have generated a certificate that contains both a RSA and ECDSA certificate, with at least 2048 bits for RSA (or more), and 256 bits for ECDSA.

---

For a cipher suite issues

Try an empty Cipher Suite exclusion list on the Jetty side.

⚠️ WARNING: This lets you use KNOWN vulnerable Cipher Suites!

1. Enable SSL/TLS for WebSocket Client
2. Blank out the Cipher Suite Exclusion List

Example (assuming Jetty 9.4.19.v20190610 or newer):

SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setExcludeCipherSuites(); // blank out the default excluded cipher suites
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client

⚠️ WARNING: This is not recommended, and any modern computer (even cell phones) can easily read your encrypted traffic