How can an openssl client trust a self signed certificate for a server

Question

I'm setting up an c++ class for handling tls connections (client and server). It works except for the tls handshake : I have generated my self signed root certificate and signed the rsa server key with it. but i get a client error which is unknown CA

script to generate self signed CA (CA file and CA.pem file)

openssl req -x509 -sha256 -days 3650 -newkey rsa:4096 -keyout CA -out CA.pem

script to generate and sign the server key (key file and key.pem file)

read -p "key and cert name :" x

openssl genrsa -out $(echo $x) 2048
openssl req -new -key $(echo $x) -out $(echo $x).csr
openssl x509 -req -in $(echo $x).csr -CA CA/CA.pem -CAkey CA/CA -CAcreateserial -out $(echo $x).pem -days 3650 -sha256

then I pass CA.pem to client using SSL_CTX_use_certificate_file , key to server using SSL_CTX_use_PrivateKey_file and key.pem using SSL_CTX_use_certificate_file

client is in mode SSL_VERIFY_PEER and server is in mode SSL_VERIFY_NONE so only client checks server certificate.

As the server key is signed using CA and client trust CA.pem it should be working but when handshake is negociated, i get this in wireshark (a message from client to server) : Alert level Fatal, Description : Unknown CA

Answer 1

If you read OpenSSL's documentation, for a client SSL_CTX_use_certificate_file installs a client certificate. It does not specify the list of trusted CAs that may be used to verify a cert.

For that, on the client side, you want to use SSL_CTX_load_verify_locations :

SSL_CTX_load_verify_locations() specifies the locations for ctx, at which CA certificates for verification purposes are located.