TLS：如何验证使用RSA密钥自签名的证书

Question

I am trying to use OpenSSL and TLS to provide SSH-like trust model. That means that the two peers have an RSA key pair stored each one, no certificates. They have also exchanged their public keys prior to the TLS session establishment. To achieve this I am using some OpenSSL's undocumented functions so I still have doubts, here is how:

• During initialisation peers generate an in-memory, temporary x509 certificate. It contains the minimal dummy data I could get away with, ie a dummy CN, issuer, and many years before expiring. These are never checked anyway . SSL context is setup to exchange both certificates both ways.

• The important step is that the certificate contains the host's public key and it is signed using the private key:

   EVP_PKEY *pkey = EVP_PKEY_new(); EVP_PKEY_assign_RSA(pkey, PRIVKEY); X509_set_pubkey(x509, pkey); X509_sign(x509, pkey, EVP_sha384()); 

• This certificate is assigned to SSL_CTX and used for the whole application lifetime.

What I'm mostly worried about is the verification process when the TLS session is established. I'm disabling all verification performed by OpenSSL using SSL_CTX_set_cert_verify_callback(always_true_func) and roll my own like this:

X509 *received_cert = SSL_get_peer_certificate(conn_info->ssl);
EVP_PKEY *received_pubkey = X509_get_pubkey(received_cert);
if (EVP_PKEY_type(received_pubkey->type) != EVP_PKEY_RSA)
    error();
ret = X509_verify(received_cert, received_pubkey);
if (ret <= 0)
    error("trust_failed");

// Compare received public key with expected one
RSA *expected_rsa_key = read_RSA_key_from_disk();
EVP_PKEY expected_pubkey = { 0 };
EVP_PKEY_assign_RSA(&expected_pubkey, expected_rsa_key);
EVP_PKEY_cmp(received_pubkey, &expected_pubkey);

if (ret == 1)
    return true; // identity verified!
else
    return false;

The question: Is this proper usage of the OpenSSL API? Do you see any security holes, especially on the last part, verification of the received key? Any better way to achieve the same result?

EDIT: The answer: To verify that a self-signed certificate received during TLS handshake matches the stored RSA key, there is no need to check the certificate's signature , ie there is no need for X509_verify . The comparison of the received public key to the expected one is enough.

The reason is that (quoting Dr Stephen Henson, OpenSSL project core developer) " Depending on the ciphersuite either an RSA decryption operation or an RSA signature operation is performed by the server. So if the handshake completes successfully you can be sure that the same key is used as the one present in the certificate ".

Answer 1

Certificates bind a public key and additional information (such as a identifier and other attributes) together. What makes this association between the public key and the additional information a certificate is the fact it is signed.

The reason X.509 certificates are signed and issued by a CA is because the CA asserts the binding between the public key and the rest of the content of the certificate (in particular its subject). The purpose of this is to let a party that knows the CA but doesn't necessarily know the entity to which the certificate was issued verify that the content of the certificate is true, in particular that the public key belongs to the certificate's subject.

Since your authentication scheme relies on pre-established knowledge of who or what owns the public keys anyway, and since you'll ignore the content of the certificate (besides the public key), there is no point verifying that the association between the public key and the rest of the certificate is true (whether self-signed or not).

What you must verify is that the public key you're receiving matches one of the public keys you already know.

EDIT:

But second is to actually verify that the self-signed certificate is validly signed, since anyone can send me the host's public key (it's public...).

Anyone can send you a certificate with the host's public key, but only the entity with the private key matching that public key will be able to negotiate the master secret. How this is done depends on the cipher suite, but only the server that has the private key will be able to carry on with the TLS connection.

The client will at least always at least know that it has established a TLS connection with the entity that has the private key for the public key in the server certificate (independently of knowing who that cert belongs to).

This really is completely independent from the mechanism used to verify the identity of the server (traditionally, PKI + host name verification).

If you know the host's public key with certainty (eg by looking it up in a list you already know), verifying the certificate signature is irrelevant (whether it's self-signed or signed by a party you don't know).

The same applies for client certificates: the handshake will not complete if the client that sent its certificate is unable to send the correct signature (in the Certificate Verify message) made using the private key matching the client certificate it has sent.