TLS server doesn't receive certificate from client, fails to terminate handshake

Question

I have a client and a server attempting to mutually authenticate one another and initiate a TLS connection. The certificates I'm using right now are self-signed.

In the server code I have SSL_VERIFY_FAIL_IF_NO_PEER_CERT set. The handshake succeeds, but SSL_get_peer_certificate returns a NULL pointer on the server side. If the client didn't return a certificate, why did the handshake not fail?

If I comment out the SSL_get_peer_certificate check on the server side, the client and server do connect and are able to communicate, but it isn't a TLS connection. When I watch them exchange packets over wireshark, I only see TCP traffic.

Server code:

BIO *acceptTLSConnection(char *port) {

  BIO *sbio, *bbio, *acpt = NULL;
  SSL_CTX *ctx = NULL;
  SSL *ssl = NULL;

  SSL_library_init();

  ctx = SSL_CTX_new(TLSv1_server_method());

  SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER|SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);

  if(!SSL_CTX_use_certificate_file(ctx,"servercert.pem",SSL_FILETYPE_PEM)
    || !SSL_CTX_use_PrivateKey_file(ctx,"serverkey.pem",SSL_FILETYPE_PEM)
    || !SSL_CTX_check_private_key(ctx)) {
    ERR_print_errors_fp(stderr);
    fatalError("Error setting up SSL_CTX.");
  }

  if(!SSL_CTX_load_verify_locations(ctx, "clientcert.pem", NULL))
    fatalError("Could not load trusted CA certificates.");

  sbio=BIO_new_ssl(ctx,0);

  BIO_get_ssl(sbio, &ssl);

  if(!ssl) {
    fatalError("Can't locate BIO SSL pointer.");
  }

  SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);

  bbio = BIO_new(BIO_f_buffer());
  sbio = BIO_push(bbio, sbio);

  acpt=BIO_new_accept(port);
  BIO_set_accept_bios(acpt,sbio);

  /* Setup accept BIO */
  if(BIO_do_accept(acpt) <= 0) {
    ERR_print_errors_fp(stderr);
    fatalError("Error in setting up accept BIO");
  }

  /* Now wait for incoming connection */
  if(BIO_do_accept(acpt) <= 0) {
    ERR_print_errors_fp(stderr);
    fatalError("Error in connection");
  }

  sbio = BIO_pop(acpt);
  BIO_free_all(acpt);

  if(BIO_do_handshake(sbio) <= 0) {
    ERR_print_errors_fp(stderr);
    fatalError("Error in SSL handshake");
  }

  /* Verify a client certificate was presented during the negotiation */
  X509* cert = SSL_get_peer_certificate(ssl);
  if(cert) { X509_free(cert); } /* Free immediately */
  if(NULL == cert) fatalError("Client did not present a cert during handshake.");

  /* Verify the result of chain verification */
  int res = SSL_get_verify_result(ssl);
  if(!(X509_V_OK == res)) fatalError("Cert presented by client couldn't be verified.");

  return sbio;
}

Client code:

BIO *makeTLSConnection(char *servIP, char *servPort) {

  char *servLoc = calloc(strlen(servIP) + strlen(servPort) + 2, sizeof(char));
  strcat(servLoc, servIP);
  strcat(servLoc, ":");
  strcat(servLoc, servPort);

  BIO *sbio = NULL;
  SSL_CTX *ctx = NULL;
  SSL *ssl = NULL;

  SSL_library_init();

  ctx = SSL_CTX_new(TLSv1_client_method());

  SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);

  if(!SSL_CTX_use_certificate_file(ctx,"clientcert.pem",SSL_FILETYPE_PEM)
    || !SSL_CTX_use_PrivateKey_file(ctx,"clientkey.pem",SSL_FILETYPE_PEM)
    || !SSL_CTX_check_private_key(ctx)) {
    ERR_print_errors_fp(stderr);
    fatalError("Error setting up SSL_CTX.");
  }

  if(!SSL_CTX_load_verify_locations(ctx, "servercert.pem", NULL))
    fatalError("Could not load trusted CA certificates.");

  sbio = BIO_new_ssl_connect(ctx);

  BIO_get_ssl(sbio, &ssl);

  if(!ssl) {
    fatalError("Can't locate SSL pointer.");
  }

  BIO_set_conn_hostname(sbio, servLoc);

  if(BIO_do_connect(sbio) <= 0) {
    ERR_print_errors_fp(stderr);
    fatalError("Error connecting to server.");
  }

  if(BIO_do_handshake(sbio) <= 0) {
    ERR_print_errors_fp(stderr);
    fatalError("Error establishing SSL connection.");
  }

  /* Verify a server certificate was presented during the negotiation */
  X509* cert = SSL_get_peer_certificate(ssl);
  if(cert) { X509_free(cert); } /* Free immediately */
  if(NULL == cert) fatalError("Server did not present a cert during handshake.");

  /* Verify the result of chain verification */
  int res = SSL_get_verify_result(ssl);
  if(!(X509_V_OK == res)) fatalError("Cert presented by server couldn't be verified.");

  return sbio;
}

Answer 1

It looks like you are missing the call to SSL_CTX_use_certificate_chain_file and SSL_CTX_set_client_CA_list on the server. I believe SSL_CTX_set_client_CA_list triggers the machinery to perform client authentication (ie, it elicits the ClientCertificate message from the RFC in section 7.4.6 during the exchange).

If SSL_CTX_set_client_CA_list causes the failure you are looking for, then I'm inclined to believe its a bug in the OpenSSL library. You should get a failure when specifying SSL_VERIFY_PEER and SSL_VERIFY_FAIL_IF_NO_PEER_CERT at the server because that's what the docs say.

Also see Is verification supposed to fail with SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT without SSL_CTX_set_client_CA_list on the OpenSSL Users mailing list. Its in response to this question.

---

Also see Testing SSL/TLS Client Authentication with OpenSSL and OpenSSL client not sending client certificate .