openSSL: handshake failure - server cannot get client certificate

Question

I am trying to write client and server code to do SSL handshake using openSSL API.

Client code contains:

 // Part of client code:
 SSL_CTX_set_verify(ctx,SSL_VERIFY_PEER,NULL);
 SSL_CTX_load_verify_locations(ctx,"ca.pem",NULL);
 ...
 if (SSL_CTX_use_certificate_file(ctx, "cli.crt" , SSL_FILETYPE_PEM) <= 0) {
  exit(1);
  }

  if (SSL_CTX_use_PrivateKey_file(ctx, "cli.key", SSL_FILETYPE_PEM) <= 0) {
  exit(1);
  }
 ...
 sd = socket (AF_INET, SOCK_STREAM, 0);
 sa.sin_family      = AF_INET;
 sa.sin_addr.s_addr = inet_addr ("127.0.0.1"); 
 sa.sin_port        = htons (44444);
 ...
 ssl = SSL_new (ctx);              
 SSL_set_fd (ssl, sd);
 err = SSL_connect (ssl); 

Server code contains:

 // Part of server code:
 SSL_CTX_set_verify(ctx,SSL_VERIFY_PEER,NULL);
 SSL_CTX_load_verify_locations(ctx,"ca.pem",NULL);
 ...
 if (SSL_CTX_use_certificate_file(ctx, "serv.crt", SSL_FILETYPE_PEM) <= 0) {
    exit(1);
 }
 if (SSL_CTX_use_PrivateKey_file(ctx, "serv.key", SSL_FILETYPE_PEM) <= 0) {
    exit(1);
  }
 ...
 listen_sd = socket (AF_INET, SOCK_STREAM, 0);  
 memset (&sa_serv, '\0', sizeof(sa_serv));
 sa_serv.sin_family      = AF_INET;
 sa_serv.sin_addr.s_addr = INADDR_ANY;
 sa_serv.sin_port        = htons (44444);

 err = bind(listen_sd, (struct sockaddr*) &sa_serv, sizeof (sa_serv));                           
 err = listen (listen_sd, 5);       
 client_len = sizeof(sa_cli);
 sd = accept (listen_sd, (struct sockaddr*) &sa_cli, &client_len);
 close (listen_sd);
 ....
 ssl = SSL_new (ctx);                           
 SSL_set_fd (ssl, sd);
 err = SSL_accept (ssl);   

I ran the code using destination address as loop back address (127.0.0.1), as can be seen from the code shown above. The client and server programs executed on same machine worked fine.

But, when I run the client and server programs on different machines (VMWare VMs - Ubuntu Linux), the code fails.

 Client VM IP:192.168.181.188
 Server VM IP:192.168.181.180

Using the IP of server VM (eg 192.168.181.180) as address in client code, I get the following error at server:

 140890B2: SSL3_GET_CLIENT_CERTIFICATE:no certificate returned: s3_srvr.c:2602:

I have created my own CA on my Linux machine and shared the CA public key file with both client and server VMs. The client and server certificates are signed by this CA.

 CA : CA123
 Client CN: Client (signed by CA123)
 Server CN: Server (signed by CA123)

The client is able to verify server certificate (I can even get server certificate and check CN is indeed 'Server'), but server can't get client certificate,so handshake fails.

Can anyone suggest a solution to this problem?

Thanks a lot.

Answer 1

Check subject of the client certificate. Some times the CN (common name) field should have IP or host name of the client host.

In your case, the client certificate could have the following subject:

Subject: C=xxx, ST=xxx, L=xxx, O=xxx, OU=xxx, CN=192.168.181.180

I am not 100% sure about this fix. But before breaking the wall, try it.

Answer 2

Okay ... I am not sure that the following is a complete solution, its just a fix that a found for my current project, I am still not satisfied with my answer.

So, by reading my question, you will observe that I mentioned that the CA was made on a linux machine and certificates for client and server were signed by that CA. That host is another VMWare VM, call it "VM3". So certificates I generated using "CA123" on "VM3" for client and server didn't work on client VM (call it "VM1") and server VM(call it "VM2"). Out of curiosity, I tried to setup a CA on the VM2, and sign client and server certificates, and copied client certificate to client VM. To my surprise, it works, but I don't know why. I can't answer what kind of dependency did I create on VM3 ?

Feel free to post a more accurate and complete answer.