Getting SSL: CERTIFICATE_VERIFY_FAILED when using python with Apache NIFI rest api

Question

I have various python scripts that use the NIFI rest api to make some calls. Locally on my machine and other's local machine's the scripts work. I am trying to run the scripts with through Jenkins. Jenkins is running using on an AWS EC2 instance. The scripts do not work on the Jenkins EC2 instance however they will work on other EC2 instances within the same AWS account and security group. The only way I am able to get the script to work on the Jenkins EC2 instance is using (verify=False) for the rest call. However I need to be able to get it working on Jenkins without (verify=False) given that some of the rest calls I need to make won't work with it.

The certs I am using are two pem files generated from a p12 file we use for NIFI. The certs work everywhere else so I do not think it is an issue with them. I have also tried various Python versions and I still get the same result so I do not think it is that either. I have the public and private ip addresses of the Jenkins server opened up for ports 22, 8443, 18443, 443, 9999, 8080, 18080. So I don't think it is a port issue either. I don't have much experience with SSL so I'm lost on what to ever try next. But given that it works locally and works on the AWS EC2 instance we're running the NIFI dev version on I am out of ideas.

Python Script (the other scripts have the same issue and similar structure):

import json, requests, sys

with open("jenkinsCerts.dat") as props:
certData = json.load(props)
cert = (certData["crt"],certData["key"])

 def makeRestGetCall(url):
#calls a RESTful url and returns the response in json format
if "https" in url:
    response = requests.get(url, cert=cert)
else:
    response = requests.get(url)
print response

with open('servers.txt') as nifi_server_list:
errorCount=0
data = json.load(nifi_server_list)

for server in data:
    try:
        print "trying: "+server["name"]+" ("+server["url"]+")"
        makeRestGetCall(server["url"], verify=False)
    except:
        print server["name"]+" ("+server["url"]+") did not respond"
        errorCount = errorCount + 1
try:
    assert errorCount==0
except AssertionError:
    print errorCount, " servers did not respond"

The script above doesn't give any error just an output that doesn't work but works on other machines at the same time.

trying: dev-cluster-node-1 dev-cluster-node-1 did not respond trying: dev-cluster-node-2 dev-cluster-node-2 did not respond trying: dev-cluster-node-3 dev-cluster-node-3 did not respond trying: dev-registry dev-registry did not respond trying: dev-standalone dev-standalone did not respond 5 servers did not respond

This is the ERROR I get from Jenkins when I run a different Python script that uses the same authentication from above but the full script was too long to copy and not necessary:

*requests.exceptions.SSLError: HTTPSConnectionPool(host='ec2-***-***-***.****-1.compute.amazonaws.com', port=8443): Max retries exceeded with url: /nifi-api/flow/process-groups/3856c256-017-****-***** (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))*

Answer 1

I believe the issue is that your script isn't aware of the expected public certificates of your NiFi servers in order to verify them during the request.

The crt and key values you are providing should contain the public certificate and private key of the Python script in order to authenticate to the NiFi servers. This material identifies the client in this case, which is required for mutual authentication TLS (one of the various authentication mechanisms NiFi supports).

However, with all TLS handshakes, the server must also provide a public certificate identifying itself and with a CN or SAN matching the hostname serving the connection (eg if you visit https://stackoverflow.com , the website needs to present a certificate issued for stackoverflow.com , not andys-fake-stackoverflow.com ).

Most websites on the public internet have their certificates signed by a Certificate Authority (Let's Encrypt, Comodo, Verisign, etc.). Your browser and many software components come with a collection of these trusted CAs so that TLS connections work out of the box. However, if the certificates used by your NiFi servers are not signed by one of these CAs, the default Python list is not going to contain their signer, so it won't be able to verify these NiFi certs. You'll need to provide the signing public certificate to your Python code to enable this.

The requests module allows you to configure a custom CA bundle path with the public certificate that signed your NiFi certs. You can obtain that in a variety of ways (you have direct access to the NiFi servers, but any connection [via browser, openssl s_client , curl , etc.] will allow you to obtain the public certificate chain). Store the public cert ( nifi_ca.pem ) in PEM format somewhere in your Python script's folder structure, and reference it like so :

response = requests.get(url, cert=cert, verify="nifi_ca.pem")