客户端在BIO_do_connect上返回-1

Question

I'm trying to create a basic server and client using OpenSSL and its BIOs but BIO_do_connect returns -1. ERR_get_error returns 0 after that. I've tried to minimize the code below by just writing // check [condition]. In my real code I'm doing the same thing with an if check and then I print out the error returned by ERR_get_error. (so if condition is true I'm printing an error msg)

This is my code for the server:

// init OpenSSL
SSL_load_error_strings();
ERR_load_BIO_strings();
SSL_library_init();
OpenSSL_add_all_algorithms();

SSL_CTX *ctx = SSL_CTX_new(SSLv23_server_method());
SSL_CTX_set_default_passwd_cb(ctx, &myPasswordCallback);

int certState = SSL_CTX_use_certificate_file(ctx, "../certs/cert.pem", SSL_FILETYPE_PEM);
// check certState < 0

int keyState = SSL_CTX_use_PrivateKey_file(ctx, "../certs/key.pem", SSL_FILETYPE_PEM);
// check keyState < 0

BIO *serverBio = BIO_new_ssl(ctx, 0);
// check serverBio == nullptr

SSL *serverSsl = nullptr;
BIO_get_ssl(serverBio, &serverSsl);
// check serverSsl == nullptr

SSL_set_mode(serverSsl, SSL_MODE_AUTO_RETRY);

BIO *acceptBio = BIO_new_accept("6672");
// check acceptBio == nullptr

int setupAcceptResult = BIO_do_accept(acceptBio);
// check setupAcceptResult <= 0

int acceptResult = BIO_do_accept(acceptBio);
// check acceptResult <= 0

BIO *clientBio = BIO_pop(acceptBio);
// check clientBio == nullptr

BIO_free_all(clientBio);

BIO_free_all(acceptBio);
BIO_free_all(serverBio);

// cleanup OpenSSL
SSL_CTX_free(ctx);
EVP_cleanup();
ERR_free_strings();

This server runs fine but my client fails to connect to it:

// init OpenSSL
SSL_load_error_strings();
ERR_load_BIO_strings();
SSL_library_init();
OpenSSL_add_all_algorithms();

SSL_CTX *ctx = SSL_CTX_new(SSLv23_client_method());
SSL_CTX_set_default_passwd_cb(ctx, &myPasswordCallback);

int certState = SSL_CTX_use_certificate_file(ctx, "../certs/cert.pem", SSL_FILETYPE_PEM);
// check certState < 0

int keyState = SSL_CTX_use_PrivateKey_file(ctx, "../certs/key.pem", SSL_FILETYPE_PEM);
// check keyState < 0

BIO *clientBio = BIO_new_ssl_connect(ctx);
SSL *clientSsl = nullptr;
BIO_get_ssl(clientBio, &clientSsl);
// check clientSsl == nullptr

SSL_set_mode(clientSsl, SSL_MODE_AUTO_RETRY);

BIO_set_conn_hostname(clientBio, "localhost:6672");

long connectionState = BIO_do_connect(clientBio);
// check connectionState <= 0
// here it fails; connectionState is -1

long sslState = SSL_get_verify_result(clientSsl);
// check sslState != X509_V_OK

BIO_free_all(clientBio);

SSL_CTX_free(ctx);

EVP_cleanup();
ERR_free_strings();

I'm sorry for posting so much code. I didn't really find a complete example of OpenSSL server/client using BIOs.

Answer 1

You server code is essentially this:

1. setup serverBio as SSL
2. create a new BIO acceptBio without SSL
3. accept the connection connection -> clientBio
4. free everything

The server is not doing any SSL handshake here since the serverBio gets not used for the newly created TCP connection clientBio.

Apart from that I recommend that you test your server and client first against known good client and server so that you can faster figure out where the problem is. openssl s_client and openssl s_server provide such test client and server. Also packet capturing (wireshark) helps to find out what happens between server and client.