SSL_read() does not return after successful handshake

Question

I made a server and a client, and they communicate in a secure TLSv1.3, OpenSSL 3.0.0 channel. They have to be able to send and receive, but in this test the server only receives, and the client only sends application data periodically.

After the successful handshake, the clients select() signals that there is data available to read. When I try to, SSL_read() does not return. I checked the network traffic in wireshark, and debugged the OpenSSL library to find out, that 2 SESSION_TICKET are sent after the handshake by the server to the client, and that is the message SSL_read() fails to process. Just for curiosity, I sent a message with the server right after the handshake to see how the client reacts to it. Surprisingly the client left its blocked state and the communication worked fine after that.

Due to the observed behavior, my only guess is that I try to read data meant for OpenSSL which is probably processed by its state machine when I call SSL_read() . When the library calls read() there is actually no application data available, so it blocks the thread.

The member function which I use to read "count" bytes can be seen below.

int EventHandler::readByte(CommBuffer &buffer, ssize_t count, struct timeval &tmout){
  fd_set rfds;
  int retSelect, retRead;
  int readByte=0;
  int endRead=0;
  int readSize=count;

  buffer.setBufferSize(count+1);
  do{
    FD_ZERO(&rfds);
    FD_SET(fd,&rfds);
    retSelect=select(fd+1,&rfds,NULL,NULL,&tmout);
    if(retSelect>0){
      if (encryptionOn) {
        retRead = SSL_read(ssl, (const_cast<char*>(&buffer.get()[readByte])), readSize); // Thread blocks here.
      } else {
        retRead = read(fd, (const_cast<char*>(&buffer.get()[readByte])), readSize);
      }
      if( retRead > 0 ){
        readByte+=retRead;
        allReadBytes+=retRead;
        buffer.setDataSize(readByte);
        if(readByte>=count) endRead=1;
        readSize=count-readByte;
      }
      else{
        if( retRead == -1 && errno == EINTR  ){
          continue;
        }
        if(fd>=0){
          close(fd);
        }
        fd=CLOSED_FD;
        return (CLOSED_FD);
      }
    }
    else if(retSelect==0){
      // handle timeout error
      return(TIMEOUT_FD);
    }
    else{
      return(retSelect);
    }
  }while(endRead!=1);
  return(readByte);
} 

Note that this function works just fine without encryption. ( read() instead of SSL_read() )

• What am I doing wrong?
• How should I read data?
• How would you modify this function?

Answer 1

select works at the TCP level while SSL_read works at the TLS level. SSL_read on a blocking socket will (usually) only return when application data are available while select will signal when any kind of data are available on the socket, ie also for non-application records or incomplete SSL frames.

In your specific case these data are the session tickets which with TLS 1.3 are no longer send as part of the TLS handshake but after the handshake. But, with lower TLS versions it could also happen that SSL_read blocked even if select returned that data are available: application data in TLS are send within SSL frames and it could happen that select signaled available data even though only part of the SSL frame is available. In this case SSL_read would block until the full SSL frame could be read. Also, SSL_read would block on SSL renegotiations even if select would signal available data.

Moreover, it could happen that select would not signal available data but SSL_read would return some. This is the case when the previous SSL_read did not consume all data which were included in the last SSL frames already read from the TCP socket.

In other words: select should not be used with blocking sockets in SSL. Instead the sockets should be made non-blocking so that SSL_read (and other functions) can fail with SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE which one should handle as documented. Additionally SSL_pending should be called to check for data which were were already read from the TCP socket but which were not yet consumed by SSL_read .

Answer 2

If you are using a blocking socket, it may be possible that mode SSL_MODE_AUTO_RETRY is enabled. this may cause a hang on the SSL_read in case there's no application data to be read. In this case, you can either disable this mode (but you will need to handle an SSL_ERROR_WANT_READ error), or use non-blocking sockets. I suggest you You can read about SSL_MODE_AUTO_RETRY .