Inconsistent IOError exception on HTTPS POST requests using Python 'requests' library

Question

I keep getting the following exception when trying to do a HTTPS POST using requests

This problem occurs sporadically and the same request when retried (using backoff module) goes through successfully. I don't know how to reproduce this but I can see this problem when I run a load of HTTPS POST requests.

Traceback (most recent call last):
 File \"/usr/local/lib/python2.7/dist-packages/shared/util/http_util.py\", line 71, in send_https_post_request
 response = session.post(url, cert=cert, data=data)
 File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 522, in post
 return self.request('POST', url, data=data, json=json, **kwargs)
 File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 475, in request
 resp = self.send(prep, **send_kwargs)
 File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 596, in send
 r = adapter.send(request, **kwargs)
 File \"/usr/local/lib/python2.7/dist-packages/requests/adapters.py\", line 423, in send
 timeout=timeout
 File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py\", line 595, in urlopen
 chunked=chunked)
 File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py\", line 352, in _make_request
 self._validate_conn(conn)
 File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py\", line 831, in _validate_conn
 conn.connect()
 File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py\", line 289, in connect
 ssl_version=resolved_ssl_version)
 File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py\", line 306, in ssl_wrap_socket
 context.load_cert_chain(certfile, keyfile)
IOError: [Errno 2] No such file or directory

Relevant Code:

@contextlib.contextmanager
def pem_bytes_as_cert_file(pem_cert_bytes):
    '''
     Given bytes, return a temporary file which can be used as the cert
    '''
    with tempfile.NamedTemporaryFile(delete=True, suffix='.pem') as t_pem:
        f_pem = open(t_pem.name, 'wb')
        f_pem.write(pem_cert_bytes)
        f_pem.close()
        yield t_pem.name

def send_https_post_request(session, url, data, pem_cert_in_bytes):
    with pem_bytes_as_cert_file(pem_cert_in_bytes) as cert:
        response = session.post(url, cert=cert, data=data)
        response.raise_for_status()
        response.close()

Could you please help me understand more about this issue?

Answer 1

 ... context.load_cert_chain(certfile, keyfile) IOError: [Errno 2] No such file or directory 

The context is obviously in loading the certificate and key

 with pem_bytes_as_cert_file(pem_cert_in_bytes) as cert: response = session.post(url, cert=cert, data=data) 

You seem to have the certificate and key as string and attempt to create a temporary file to give it as cert argument

with tempfile.NamedTemporaryFile(delete=True, suffix='.pem') as t_pem:
    f_pem = open(t_pem.name, 'wb')
    f_pem.write(pem_cert_bytes)
    f_pem.close()
    yield t_pem.name

You create a temporary file with tempfile.NamedTemporaryFile and write the cert+key string into it. Then you close the temporary file. The documentation for NamedTemporaryFile states:

If delete is true (the default), the file is deleted as soon as it is closed.

Thus, the file is deleted as soon you close it. Only if you are lucky (race conditions) the file is still accessible in the system when you try to use it from inside session.post .