How to not block making initial connection with python requests?

Question

I have a server that uses requests to make queries to another server that may or may not be running. If that server is not running, I do not want to block for a long time; I can just handle the error right away. However, the timeout parameter does not seem to apply to the process of making the initial connection.

From the terminal, I run:

>>> import time
>>> import requests
>>> t1 = time.time() ; exec("try: requests.get('http://192.168.99.100/', timeout=1.0)\nexcept: pass") ; t2 = time.time() ; t2 - t1
21.00611114501953

This takes about 21 seconds and has no dependance on the timeout I give. I also tried using eventlet's timeout, but it turned out the same:

>>> import time
>>> import eventlet
>>> requests = eventlet.import_patched('requests')
>>> t1 = time.time() ; exec("try: \n  with eventlet.Timeout(1): requests.get('http://192.168.99.100/')\nexcept: pass") ; t2 = time.time() ; t2 - t1
21.00276017189026

The error I am getting for the connection is:

ConnectionError: ('Connection aborted.', error(11, 'Resource temporarily unavailable'))

Finally, I am running python under the Windows Subsystem for Linux, which might be working with sockets differently.

Answer 1

From the docs:

timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds).

That being said an example using something like polling.

import gevent, time
import requests

from gevent import monkey, spawn, sleep
monkey.patch_all() # for working with requests/sockets


do_not_wait = []
print(time.time())

do_not_wait.append(gevent.spawn(requests.get, 'http://192.168.99.100/', timeout=1.0))
do_not_wait.append(gevent.spawn(requests.get, 'http://192.168.99.100/', timeout=1.0))
do_not_wait.append(gevent.spawn(requests.get, 'http://192.168.99.100/', timeout=1.0))

[x.start() for x in do_not_wait]
print(time.time())

while True:
    sleep(1)
    print([x.ready() for x in do_not_wait])

Answer 2

You could use a task manager (like celery ) then set the timeout on the worker thread. I think requests has support for gevent not eventlet so that may be why you're seeing the same issues using eventlet .