Non-Blocking I/O With OpenSSL BIO

Question

I am using OpenSSL 1.0.0-fips on Linux. The issue I am having is that SSL_connect() is returning -1 and SSL_get_error() is returning SSL_ERROR_WANT_READ . I then put the file descriptor into a select() with a timeval structure set to 10 seconds and the select() just times out.

I fired up Wireshark and I see the "Client Hello" go out and I see the ServerHello come back to the client but it never "wakes up" in the select() . It just times out.

My questions are:

1. Do I have to create a BIO object using BIO_new_socket() and then assign the BIO object to my SSL object using SSL_set_bio() ? The man page for SSL_set_fd() says it will automatically create a BIO object so that seems to imply that SSL_set_bio() is sort of a useless function that you never really have to call.

2. Let us say we use SSL_set_fd() and assign a connected TCP file descriptor that is blocking. Let us say that we then later change that file descriptor to non-blocking using fcntl() . Does this break the SSL object (or the underlying BIO object) in anyway?

Answer 1

1) Do I have to create a BIO object using BIO_new_socket() and then assign the BIO object to my SSL object using SSL_set_bio()? The man page for SSL_set_fd() says it will automatically create a BIO object so that seems to imply that SSL_set_bio() is sort of a useless function that you never really have to call.

If the default BIO object is sufficient for your needs, then you do not have to manually create and install your own BIO object. The SSL_set_bio() call is there just in case you'd like to create/use a BIO object that is different from the default one that SSL_set_fd() creates on your behalf.

2) Let us say we use SSL_set_fd() and assign a connected TCP file descriptor that is blocking. Let us say that we then later change that file descriptor to non-blocking using fcntl(). Does this break the SSL object (or the underlying BIO object) in any way?

Yes, I believe it will break. The calling patterns for non-blocking OpenSSL are very different from those used with blocking OpenSSL, and I don't believe you can just switch back and forth from one mode to the other on the fly. That said, I haven't tried it myself, so I could be wrong, but I think to be on the safe side (and to be consistent) you should choose up-front whether you want to use blocking or non-blocking I/O and stick with it for the duration of the connection.

In particular, this quote from the man page:

The BIO and hence the SSL engine inherit the behaviour of fd.

... suggest that the SSL setup calls will examine the state of your fd and set private variables within the BIO and the SSL objects based on the blocking/non-blocking state of the fd. If you then "go behind OpenSSL's back" and change the behavior of the fd, OpenSSL's routines will not expect that and will very likely do the wrong thing and not work correctly.