Deadlock with PyMongo and gevent

Question

I am using PyMongo and gevent together, from a Django application. In production, it is hosted on Gunicorn.

I am creating a single Connection object at startup of my application. I have some background task running continuously and performing a database operation every few seconds.

The application also serves HTTP requests as any Django app.

The problem I have is the following. It only happens in production, I have not been able to reproduce it on my dev environment. When I let the application idle for a little while (although the background task is still running), on the first HTTP request (actually the first few), the first "find" operation I perform never completes. The greenlet actually never resumes. This causes the first few HTTP requests to time-out.

How can I fix that? Is that a bug in gevent and/or PyMongo?

Answer 1

I found what the problem is. By default PyMongo has no network timeout defined on the connections, so what was happening is that the connections in the pool got disconnected (because they aren't used for a while). Then when I try to reuse a connection and perform a "find", it takes a very long time for the connection be detected as dead (something like 15 minutes). When the connection is detected as dead, the "find" call finally throws an AutoReconnectError, and a new connection is spawned up to replace to stale one.

The solution is to set a small network timeout (15 seconds), so that the call to "find" blocks the greenlet for 15 seconds, raises an AutoReconnectError, and when the "find" is retried, it gets a new connection, and the operation succeeds.