python ssl (eqivalent of openssl s_client -showcerts ) How to get list of CAs for client certs from server

Question

I have a group of nginx servers, that accept client certificates. They have the ssl_client_certificate option with a file containing one or more CAs

If I use a web browser, then the web browser seems to receive a list of valid CAs for client certs. The browser shows only client certs signed by one of these CAs.

Following openssl command gives me a list of CA certs:

openssl s_client -showcerts -servername myserver.com -connect myserver.com:443 </dev/null

The lines I am interested in look following way:

---
Acceptable client certificate CA names
/C=XX/O=XX XXXX
/C=YY/O=Y/OU=YY YYYYYL
...
Client Certificate Types: RSA sign, DSA sign, ECDSA sign

How can I get the same information with python?

I do have following code snippet, that allows to obtain a server's certificate, but this code does not return the list of CAs for client certs.

import ssl

def get_server_cert(hostname, port):
    conn = ssl.create_connection((hostname, port))
    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    sock = context.wrap_socket(conn, server_hostname=hostname)
    cert = sock.getpeercert(True)
    cert = ssl.DER_cert_to_PEM_cert(cert)
    return cerft

I expected to find a functional equivalent of getpeercert() , something like getpeercas() but didn't find anything.

Current workaround:

import os
import subprocess


def get_client_cert_cas(hostname, port):
    """
    returns a list of CAs, for which client certs are accepted
    """

    cmd = [
        "openssl",
        "s_client",
        "-showcerts",
        "-servername",  hostname,
        "-connect",  hostname + ":" + str(port),
        ]

    stdin = open(os.devnull, "r")
    stderr = open(os.devnull, "w")

    output = subprocess.check_output(cmd, stdin=stdin, stderr=stderr)
    ca_signatures = []
    state = 0
    for line in output.decode().split("\n"):
        print(state, line)
        if state == 0:
            if line == "Acceptable client certificate CA names":
                state = 1
        elif state == 1:
            if line.startswith("Client Certificate Types:"):
                break
            ca_signatures.append(line)
    return ca_signatures

Update:Solution with pyopenssl (Thanks Steffen Ullrich)

@Steffen Ulrich suggested to use pyopenssl, which has a method get_client_ca_list() and this helped me to write a small code snippet.

Below code seems to work. Not sure if it can be improved or whether there are any pit falls.

If nobody is answering within the next days I will post this as a potential answer.

import socket
from OpenSSL import SSL

def get_client_cert_cas(hostname, port):
    ctx = SSL.Context(SSL.SSLv23_METHOD)
    # If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
    # an empty result
    ctx.set_options(SSL.OP_NO_TLSv1_3)
    sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
    # next line for SNI
    sock.set_tlsext_host_name(hostname.encode("utf-8"))
    sock.connect((hostname, port))
    # without handshake get_client_ca_list will be empty
    sock.do_handshake()  
    return sock.get_client_ca_list()

Update: 2021-03-31

Above suggested solution using pyopenssl works in most cases. However sock.get_client_ca_list()) cannot be called immediately after performing a sock.connect((hostname, port)) Some actions seem to be required in between these two commands.

Initially I used sock.send(b"G") , but now I use sock.do_handshake() , which seems a little cleaner.

Even stranger, the solution doesn't work with TLSv1.3 so I had to exclude it.

Answer 1

With @Steffen Ullrich's help I found following solution, which works for most servers.

However I encountered situations, where the solution is not working Thus I remove the "correct" answer check until the solution works with all https servers

It requires to install an external package

pip install pyopenssl

Then following work will work:

import socket
from OpenSSL import SSL

def get_client_cert_cas(hostname, port):
    ctx = SSL.Context(SSL.SSLv23_METHOD)
    # If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
    # an empty result
    ctx.set_options(SSL.OP_NO_TLSv1_3)
    sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
    # next line for SNI
    sock.set_tlsext_host_name(hostname.encode("utf-8"))
    sock.connect((hostname, port))

    # without handshake get_client_ca_list will be empty
    sock.do_handshake()  
    return sock.get_client_ca_list()

The line sock.do_handshake() is required to trigger enough of the SSL protocol. Otherwise client_ca_list information doesn't seem to be populated.

At least for the servers, that I tested I had to make sure TLSv1.3 is not used. I don't know whether this is a bug, a feature or whether with TLSv1.3 another function has to be called prior to calling get_client_ca_list()

I am no pyopenssl expert, but could imagine, that there is a more elegant / more explicit way to get the same behavior.

but so far this works for me for all the servers, that I encountered.

Answer 2

As a generic example in python

1. first you need to contact the server to learn which issuer CA subjects it accepts:

from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509Name
from certifi import where
import idna


def get_server_expected_client_subjects(host :str, port :int = 443) -> list[X509Name]:
    expected_subjects = []
    ctx = SSL.Context(method=SSL.SSLv23_METHOD)
    ctx.verify_mode = SSL.VERIFY_NONE
    ctx.check_hostname = False
    conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
    conn.connect((host, port))
    conn.settimeout(3)
    conn.set_tlsext_host_name(idna.encode(host))
    conn.setblocking(1)
    conn.set_connect_state()
    try:
        conn.do_handshake()
        expected_subjects :list[X509Name] = conn.get_client_ca_list()
    except SSL.Error as err:
        raise SSL.Error from err
    finally:
        conn.close()
    return expected_subjects

This did not have the client certificate, so the TLS connection would fail. There are a lot of bad practices here, but unfortunately they are necessary and the only way to gather the message from the server before we actually want to attempt client authentication using hte correct certificate.

2. Next you load the cert based on the server:

from pathlib import Path
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
from pathlib import Path

def check_client_cert_issuer(client_pem :str, expected_subjects :list) -> str:
    client_cert = None
    if len(expected_subjects) > 0:
        client_cert_path = Path(client_pem)
        cert = load_certificate(FILETYPE_PEM, client_cert_path.read_bytes())
        issuer_subject = cert.get_issuer()
        for check in expected_subjects:
            if issuer_subject.commonName == check.commonName:
                client_cert = client_pem
                break
    if client_cert is None or not isinstance(client_cert, str):
        raise Exception('X509_V_ERR_SUBJECT_ISSUER_MISMATCH') # OpenSSL error code 29
    return client_cert

In a real app (not an example snippet) you would have a database of some sort to take the server subject and lookup the location of the cert to load - this example does it in reverse for demonstration only.

3. Make the TLS connection, and capture any OpenSSL errors:

from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509, FILETYPE_PEM
from certifi import where
import idna


def openssl_verifier(conn :SSL.Connection, server_cert :X509, errno :int, depth :int, preverify_ok :int):
    ok = 1
    verifier_errors = conn.get_app_data()
    if not isinstance(verifier_errors, list):
        verifier_errors = []
    if errno in OPENSSL_CODES.keys():
        ok = 0
        verifier_errors.append((server_cert, OPENSSL_CODES[errno]))
    conn.set_app_data(verifier_errors)
    return ok

client_pem = '/path/to/client.pem'
client_issuer_ca = '/path/to/ca.pem'
host = 'example.com'
port = 443

ctx = SSL.Context(method=SSL.SSLv23_METHOD) # will negotiate TLS1.3 or lower protocol, what every is highest possible during negotiation
ctx.load_verify_locations(cafile=where())
if client_pem is not None:
    ctx.use_certificate_file(certfile=client_pem, filetype=FILETYPE_PEM)
    if client_issuer_ca is not None:
        ctx.load_client_ca(cafile=client_issuer_ca)
ctx.set_verify(SSL.VERIFY_NONE, openssl_verifier)
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
    conn.do_handshake()
    verifier_errors = conn.get_app_data()
except SSL.Error as err:
    raise SSL.Error from err
finally:
    conn.close()

# handle your errors in your main app
print(verifier_errors)

Just make sure you handle those OPENSSL_CODES errors if any are encountered, the lookup dictionary is here .

Many validations occur pre verification inside OpenSSL itself and all PyOpenSSL will do is a basic validation. so we need to access these codes from OpenSSL if we want to do Client Authentication, ie on the client and throw away the response from an untrusted server if it fails any authentication checks on the client side, per Client Authorisation or rather mutual-TLS dictates