Decrypt bytes received from a websocket client on tls before the handshake

Question

I have developed a sockets server compatible with websockets to communicate applications made in java and web. But the latest Chrome and Mozilla updates no longer allow insecure connections to websockets. Then I am forced to decrypt the bytes that my server receives before proceeding with the handshake and the rest of the protocol https://www.rfc-editor.org/rfc/rfc6455

I have achieved the following:

1. Obtain the public key from a certificate signed by a CA. And the private key of my server

2. With the Cipher class of Java I have managed to use these keys to encrypt and decrypt a test string

But what I still can not do is decrypt the bytes I receive from the websocket client before proceeding with the handshake.

I hope you can help me. Thank you

The error I receive: Data must not be longer than 256 bytes

Answer 1

Solved! The decryption was entrusted to the SSLSocket class. In case someone wants to do it here the steps.

Export the certificates issued by the CA and the private key to a p12 file

openssl pkcs12 -export -in certificate/path/certificate.crt -inkey /path/privatekey/private.key -out filep12.p12 -name your_domain -CAfile /path/ca.crt -caname your_ca

Java key store

keytool -genkey -alias your_alias -keyalg RSA -keystore name_store.jks -keysize 2048

After entering a password(your_password) and confirm

keytool -importkeystore -srckeystore name_store.jks -destkeystore name_store.jks -deststoretype pkcs12 -srcstorepass your_password

keytool -delete -alias your_alias -keystore name_store.jks -storepass your_password

keytool -importkeystore -deststorepass your_password -destkeypass your_password -destkeystore name_store.jks -srckeystore filep12.p12 -srcstoretype PKCS12 -srcstorepass your_password -alias your_domain

your_alias must not be the same or similar to your_domain, the password is asked to enter (your_password) in each step that is always the same so that when decrypting there are no padding errors

The class in java

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLServerSocketFactory;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.TrustManagerFactory;


public class SServidor {   


public SServidor(){
    try {
        KeyStore keystore = KeyStore.getInstance(KeyStore.getDefaultType());
        File keystrorefile = new File("/path/name_store.jks");
        System.out.println(keystrorefile.getAbsolutePath());
        InputStream keystoreStream = new FileInputStream(keystrorefile);
        char[] passphrase="your_password".toCharArray();            

        keystore.load(keystoreStream, passphrase);
        KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        keyManagerFactory.init(keystore, passphrase);
        makeSSLSocketFactory(keystore, keyManagerFactory);

    } catch (KeyStoreException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (FileNotFoundException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (NoSuchAlgorithmException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (IOException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (CertificateException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (UnrecoverableKeyException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    }
}

private void log(Object msj){
    System.out.println(msj.toString());
}

public void makeSSLSocketFactory(KeyStore loadedKeyStore, KeyManagerFactory key){

    try {
        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustManagerFactory.init(loadedKeyStore);
        SSLContext ctx = SSLContext.getInstance("TLS");            
        ctx.init(key.getKeyManagers(), trustManagerFactory.getTrustManagers(), null);
        SSLServerSocketFactory sslssf = ctx.getServerSocketFactory();
        ServerSocket conexion = sslssf.createServerSocket(your_port);
        SSLSocket cliente=(SSLSocket) conexion.accept();

        cliente.startHandshake();
        InputStream in = cliente.getInputStream();            
        OutputStream out = cliente.getOutputStream();
        int byte_recibido=-1;   

        while(cliente.isConnected() && (byte_recibido=in.read())>-1){
            Integer n=byte_recibido & 0xFF;  
            String s=new String(String.valueOf(Character.toChars(n)));
            log(s);
        }

            out.close();                        
            bin.close();
                    in.close();
            cliente.close();
            conexion.close();


    } catch (NoSuchAlgorithmException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (KeyStoreException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (KeyManagementException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    } catch (IOException ex) {
        Logger.getLogger(SServidor.class.getName()).log(Level.SEVERE, null, ex);
    }

} 

}`

Something else, the connection to the websocket should be like this wss://your_domain:port An IP address must not be entered in the websocket url, it must be done with the domain registered in the certificate issued by the CA

With the decrypted bytes I can proceed with the RFC6455 protocol. This is only the test that I did, obviously for a sockets application, in addition, it is necessary to asynchronously handle the clients that connect to the server. I do this with the ExecutorService class but that is another topic