Python requests.post certificate verify fails

Question

I have a python app that connects to a web page (wikis.xxx.yyy) via HTTPS to retrieve and parse data from it. I can browse to the web page given that the certificate chain required to do so is in the browser (my system uses Ubuntu 14.04 LTS).

$ python --version
Python 2.7.6

The call to connect:

data = requests.post(URL, data=payload) 

is failing with the following error:

requests.exceptions.SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

I can't use verify=False given that if I do I get this error:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)

I tried passing the certificate in the verify as a client side side certificate ( http://docs.python-requests.org/en/latest/user/advanced/ ): data = requests.post(radio_status_URL, data=payload, verify="/home/user/.cert/foo.pem") but it does not connect and actually does not return any errors.

Thinking that I needed to install the certificate chain into the trusted Ubuntu CA store I followed the instructions from: https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate The foo.cert certificate that I added has the following format:

-----BEGIN CERTIFICATE-----
<....>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<....>
-----END CERTIFICATE-----

Using openssl I can see that the certificate chain is OK and that I can connect to wikis.xxx.yyy :-)

$ openssl s_client -CApath ~/.cert/ -connect wikis.xxx.yyy:443
CONNECTED(00000003)
depth=2 DC = local, DC = windows, CN = RIM Root CA MCA01YKF
verify return:1
depth=1 DC = net, DC = rim, CN = RIM Subordinate CA MCA03YKF
verify return:1
depth=0 C = CA, ST = Ontario, L = Waterloo, O = BlackBerry, OU = BTS Operations, CN = wikis.xxx.yyy, emailAddress = no-reply@xxx.yyy

verify return:1
---
Certificate chain
 0 s:/C=CA/ST=Ontario/L=Waterloo/O=BlackBerry/OU=BTS Operations/CN=wikis.xxx.yyy/emailAddress=no-reply@xxx.yyy
   i:/DC=net/DC=rim/CN=RIM Subordinate CA MCA03YKF
---
Server certificate
-----BEGIN CERTIFICATE-----
<...>
-----END CERTIFICATE-----
subject=/C=CA/ST=Ontario/L=Waterloo/O=BlackBerry/OU=BTS Operations/CN=wikis.xxx.yyy/emailAddress=no-reply@xxx.yyy
issuer=/DC=net/DC=rim/CN=RIM Subordinate CA MCA03YKF
---
No client certificate CA names sent
---
SSL handshake has read 1810 bytes and written 609 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : RC4-SHA
    Session-ID: FFD3D3EAC3807427D2D5E5BB2D70F064E2BB14218A368676A3225FBC5E5F1078
    Session-ID-ctx: 
    Master-Key: 930FBFB110BFC861E06C6A27DEBB15C2524A86F7EFC56693F4E52BA33F0AD236E5182169039909EC1BB3DE89C4C96B2B
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1442413894
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

So it works and gives a ret code of 0. Unfortunately python fails to do the same and I have no clue how to connect :-(

How do I fix this?

Answer 1

From your last testing case it shows that wikis.xxx.yyy use TLSv1.2, however Requests module needs a little revise to use it.

Please refer to this post: https://lukasa.co.uk/2013/01/Choosing_SSL_Version_In_Requests/

And there is a good plug-in to do that:https://toolbelt.readthedocs.org/en/latest/user.html#ssladapter

BTW, if you have installed urllib3, the Requests will use it, but urllib3 doesn't support TLS v1.2 until now.