ConscryptEngine data read issue : Unable to parse TLS packet header

Question

Below i the code for unwrap the data packets received from sslengine:

private ByteBuffer doUnwrap() throws IOException {
        if (mPeerNetData.position() == 0) {
            // The network input buffer is empty; read data from the channel before doing the unwrap
            final int count = mSocketChannel.read(mPeerNetData);
            Log.d(TAG, "bytesRead : " + count);
            if (count == -1) {
                handleEndOfStream(mSocketChannel, mSslEngine);
                return null;
            }
        }

        Log.d(TAG, "isReadPending :" + isReadPending);
        if (!isReadPending) {
            mPeerNetData.flip();
        }

        final SSLEngineResult result;
        try {
            result = mSslEngine.unwrap(mPeerNetData, mPeerAppData);
        } catch (SSLException e) {
            Log.d(TAG, "Exception while calling SSLEngine.unwrap()" + e);
            shutdown();
            return null;
        }
        mPeerNetData.compact();
        Log.d(TAG, "Result of SSLEngine.unwrap(): {}" + result.getStatus());


        final SSLEngineResult.Status status = result.getStatus();
        switch (status) {
            case OK:
                if (mPeerNetData.position() != 0) {
                    isReadPending = true;
                    mPeerAppData = ensureRemaining(mPeerAppData, mSslEngine.getSession().getApplicationBufferSize());
                    doUnwrap();
                }
                break;

            case CLOSED:
                closeConnection(mSocketChannel, mSslEngine);
                break;

            case BUFFER_UNDERFLOW:
                // The network input buffer might not have enough space, re-allocate if necessary
                // (NOTE: packet buffer size as reported by the SSL session might change dynamically)
                mPeerNetData = ensureRemaining(mPeerNetData, mSslEngine.getSession().getPacketBufferSize());

                // Read data from the channel, retry unwrap if not end-of-stream
                final int count = mSocketChannel.read(mPeerNetData);
                if (count == -1) {
                    handleEndOfStream(mSocketChannel, mSslEngine);
                    return null;
                }
                doUnwrap();
                break;

            case BUFFER_OVERFLOW:
                // The application input buffer does not have enough space, re-allocate and retry unwrap
                // (NOTE: application buffer size as reported by the SSL session might change dynamically)
                mPeerAppData = ensureRemaining(mPeerAppData, mSslEngine.getSession().getApplicationBufferSize());
                doUnwrap();
                break;

            default:
                throw new IllegalStateException("Invalid SSL status: " + status);
        }

        return mPeerAppData;
    }

mPeerNetData buffer has some data remains to read during unwrap and therefor i have enlarged the mPeerAppData buffer to hold the more data that has to be read from mPeerNetData buffer in next iteration.

I have debugged the flow and can verify that mPeerAppData buffer has sufficient space to hold the data and also mPeerNetData buffer has data pending to be unwrap.

But during unwrap i am getting the error has below:

javax.net.ssl.SSLException: Unable to parse TLS packet header
at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:798)
at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:738)

Please suggest how we can avoid this error?

Answer 1

After spending few days and debugging the code, i was able to know the root cause of this issue.

Above codes start working fine after i figured out that during the below operation

result = mSslEngine.unwrap(mPeerNetData, mPeerAppData);

where mPeerNetData is buffer containing the encrypted bytes read from socketChannel and mPeerAppData is a buffer holding the decrypted bytes after unwrap operation.

After the unwrap operation successfully decrypt a packet from mPeerNetData , there were still some bytes left and were not sufficient to perform unwrap action again. To do further unwrap operation, we need to read more data from socketChannel again but keep in mind we should not clear the mPeerNetData instead we should add the new data to mPeerNetData which has some bytes left previously.

Those previous left bytes are must to have in buffer for success-full decryption of data because each unwrap tries to read some initial bytes from the packet.In that way during next unwrap operation the action will success-full.

Also please note that to read complete socketChannel data you should have some while loop logic to invoke this method until you read the complete data from channel.