How do I increase the timeout for imaplib requests?

Question

I'm using imaplib to query Gmail's IMAP, but some requests are taking more than 60 seconds to return. This is already done in a task, so I have a full 10 minutes to do the request, but my tasks are failing due to the 60 second limit on urlfetch .

I've tried setting urlfetch.set_default_fetch_deadline(600) , but it doesn't seem to do anything.

Here's a stacktrace:

The API call remote_socket.Receive() took too long to respond and was cancelled.
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 760, in uid
    typ, dat = self._simple_command(name, command, *args)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 897, in _command_complete
    typ, data = self._get_tagged_response(tag)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 999, in _get_tagged_response
    self._get_response()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 916, in _get_response
    resp = self._get_line()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1009, in _get_line
    line = self.readline()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/imaplib.py", line 1171, in readline
    return self.file.readline()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/socket.py", line 445, in readline
    data = self._sock.recv(self._rbufsize)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ssl.py", line 301, in recv
    return self.read(buflen)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/ssl.py", line 220, in read
    return self._sslobj.read(len)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 864, in recv
    return self.recvfrom(buffersize, flags)[0]
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 903, in recvfrom
    apiproxy_stub_map.MakeSyncCall('remote_socket', 'Receive', request, reply)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall
    rpc.CheckSuccess()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call remote_socket.Receive() took too long to respond and was cancelled.

Answer 1

Which kind of DeadlineExceededError?

There are three kinds of DeadlineExceededError in AppEngine.
https://developers.google.com/appengine/articles/deadlineexceedederrors

1. google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests.
2. google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option.
3. google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

As you can see. The 10mins limit of taskqueue only help the google.appengine.runtime.DeadlineExceededError . The type of DeadlineExceededError can be identified via traceback (list below). In this case, it is google.appengine.runtime.apiproxy_errors.DeadlineExceededError . Which will raise in 5secs by default. (I will update the post after figure out how to change it)

TYPE 1. google.appengine.runtime.DeadlineExceededError

The traceback looks like

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~tagtooadex2/test.371204033771063679/index.py", line 9, in get
    pass
DeadlineExceededError

Solution

This exception can be solved by using taskqueue (10mins), backend or manual scaling options. https://developers.google.com/appengine/docs/python/modules/#Python_Instance_scaling_and_class

TYPE 2. google.appengine.runtime.apiproxy_errors.DeadlineExceededError

The traceback looks like

DeadlineExceededError: The API call remote_socket.Receive() took too long to respond and was cancelled.

TYPE 3. google.appengine.api.urlfetch_errors.DeadlineExceededError

The traceback looks like

DeadlineExceededError: Deadline exceeded while waiting for HTTP response from URL:     http://www.sogi.com.tw/newforum/article_list.aspx?topic_ID=6089521

Solution:

This exception can be solved by extend the deadline.

urlfetch.fetch(url, deadline=10*60)

https://developers.google.com/appengine/docs/python/urlfetch/fetchfunction

Answer 2

There's no mentioning of timeout in imaplib sources. So there are several options:

1. IMAP uses socket library to establish the connection, consider using socket.setdefaulttimeout(timeoutValue) but if you do so be aware of side-effects.
2. The second option is to create your own IMAP4 class child with tunable timeout, shall we say in open function

Answer 3

From the Google App Engine documentation , it seems like there are many possible causes for DeadlineExceededError .

In your case, it seems that it may be one of the last two (out of three) types of DeadlineExceededError presented on the page.