Mutual authentication always succeeds with OpenSSL

Question

I am using openssl and zmq to write a server and a client. My client and server need mutual authentication. but after I set SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_FAIL_IF_NO_PEER_CERT,NULL) on server, the handshake always successes whether the client send the certificate or not. In addition, SSL_get_peer_certificate(tls->get_ssl_()) return null and SSL_get_verify_result(tls->get_ssl_()) return 0 which means X509_V_OK .

I am really confused and desperate now. Any suggestions or corrections?

This is part of my code:

OpenSSL_add_all_algorithms();
SSL_library_init();
SSL_load_error_strings();
ERR_load_BIO_strings();

const SSL_METHOD *meth;
SSL_CTX *ssl_ctx;

     //**************************part of client************************
  {
    meth = SSLv23_client_method();
    ssl_ctx = SSL_CTX_new(meth);   


    SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_PEER,NULL);

    int rc1 = SSL_CTX_load_verify_locations(ssl_ctx, ".\\demoCA\\private\\server_chain.pem",".\\demoCA\\private\\");///   
     SSL_CTX_set_default_passwd_cb_userdata(ssl_ctx,"pw");

     std::string cert_chain(".\\demoCA\\private\\client_chain.pem");
     std::string cert(".\\demoCA\\private\\client_crt.pem");
     std::string key(".\\demoCA\\private\\client_key.pem");

     int code = SSL_CTX_use_certificate_chain_file(ssl_ctx,cert_chain.c_str());

     if (code != 1)
    {
         std::cout<<"error1\n";
        //throw TLSException("failed to read credentials.");
     }
    code = SSL_CTX_use_PrivateKey_file(ssl_ctx,key.c_str(),SSL_FILETYPE_PEM);   
    i f (code != 1)
    {
        std::cout<<"error2\n";
        //throw TLSException("failed to read credentials.");
    }
    if(!SSL_CTX_check_private_key(ssl_ctx))
    {
        std::cout<<"key wrong";
        system("pause");
        exit(0);
    }
   }

//*****************part of server****************************
{
    meth = SSLv23_server_method();
    ssl_ctx = SSL_CTX_new(meth);

    SSL_CTX_set_verify(ssl_ctx,SSL_VERIFY_FAIL_IF_NO_PEER_CERT,NULL)   
    SSL_CTX_set_client_CA_list(ssl_ctx,SSL_load_client_CA_file(".\\demoCA\\private\\client_chain.pem"));//

    SSL_CTX_set_default_passwd_cb_userdata(ssl_ctx,"pw");

    std::string cert_chain(".\\demoCA\\private\\server_chain.pem");
    std::string cert(".\\demoCA\\private\\server_crt.pem");
    std::string key(".\\demoCA\\private\\server_key.pem");

    int rc = SSL_CTX_use_certificate_file(ssl_ctx,cert.c_str(),SSL_FILETYPE_PEM);

    if (rc!=1)
    {
        //throw TLSException("failed to read credentials.");
        std::cout<<"error1\n";
    }

    rc = SSL_CTX_use_PrivateKey_file(ssl_ctx,key.c_str(),SSL_FILETYPE_PEM);

    if (rc!=1)
    {
        //throw TLSException("failed to read credentials.");   
        std::cout<<"error2\n";
    }

    int rcode = SSL_CTX_check_private_key(ssl_ctx);
    if(rcode!=1)
    {
        std::cout<<"key wrong";
        system("pause");
        //exit(0);
    }
}

Answer 1

From the documentation of SSL_CTX_set_verify :

SSL_VERIFY_FAIL_IF_NO_PEER_CERT

Server mode: if the client did not return a certificate, the TLS/SSL handshake is immediately terminated with a "handshake failure" alert. This flag must be used together with SSL_VERIFY_PEER.

You did not use it together with SSL_VERIFY_PEER as described in the documentation and thus it has no effect.