How can I force pymongo to close sockets?

Question

I'm currently working on distributed computing. My workers returns its results by inserting it into a mongoDB database.The code works well,but the connection remains opened and at a moment my system run out of sockets. Here is my worker code:

def worker(elt):
    client=pymongo.MongoClient(MONGODB_URI)
    db = client.get_default_database()
    essaiElt = db['essaiElt']
    #compute here
    essaiElt.insert( elt.toDict())
    client.close()

By using this command "netstat -anbo" I can see all sockets still opened (more than 3000), the max number of worker is 14 but they have to deal with more than 10 000 task.

...
TCP 10.130.151.11:4999 10.130.137.128:27017 En attente 0
TCP 10.130.151.11:5000 10.130.137.128:27017 En attente 0

I've tried to set timeouts but it doesn't have any effect.

How can I close sockets without restart my dataBase?

Python 2.7.12 Pymongo 3.3 mongoDB 3.2.10

Answer 1

What's likely happening is, you create a client, insert a document, and close the client, many times per second. A MongoClient can take a second or two to complete its shutdown process. (A MongoClient starts a background thread per server, and these threads don't exit instantly.) Even once the MongoClient has completely closed its sockets, the MongoDB server takes seconds to clean up all resources related to the TCP connection, and the OS's network layer takes minutes to clean up. (See the TIME-WAIT state in Wikipedia's TCP entry .)

Generally, you should create one MongoClient at the beginning of your Python process, and use the one MongoClient throughout that Python process lifetime:

client = pymongo.MongoClient(MONGODB_URI)

def worker(elt):    
    db = client.get_default_database()
    essaiElt = db['essaiElt']
    #compute here
    essaiElt.insert( elt.toDict())

Don't create a new MongoClient per operation. Never close it.

See also the PyMongo FAQ :

Create this client once for each process, and reuse it for all operations. It is a common mistake to create a new client for each request, which is very inefficient.