随着更多并发用户连接，Neo4j 连接错误

Question

• Neo4j 3.5.12 Community
• Python 3.6.9
• neo4j-driver==1.7.6
• neobolt==1.7.17
• Ubuntu Server 18.04
• Flask 1.1.2

A web server I run has suddenly experienced a 2000% increase in traffic. Everything was working fine before this increase. However, now I'm seeing the following error after a few hours (unsure how long, but within 5 hours) of heavy traffic and then fail constantly after that. Like connections are not being closed or something.

     db = GraphDatabase.driver("bolt://localhost:MYPORTNO", auth=basic_auth(DATABASE_USERNAME, DATABASE_PASSWORD, encrypted=False)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neo4j/__init__.py", line 120, in driver
     return Driver(uri, **config)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neo4j/__init__.py", line 161, in __new__
     return subclass(uri, **config)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neo4j/__init__.py", line 235, in __new__
     pool.release(pool.acquire())
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neobolt/direct.py", line 715, in acquire
     return self.acquire_direct(self.address)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neobolt/direct.py", line 608, in acquire_direct
     connection = self.connector(address, error_handler=self.connection_error_handler)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neo4j/__init__.py", line 232, in connector
     return connect(address, **dict(config, **kwargs))
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neobolt/direct.py", line 972, in connect
     raise last_error
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neobolt/direct.py", line 962, in connect
     s = _connect(resolved_address, **config)
   File "/home/ubuntu/api/env/lib/python3.6/site-packages/neobolt/direct.py", line 843, in _connect
     raise ServiceUnavailable("Failed to establish connection to {!r} (reason {})".format(resolved_address, error))
neobolt.exceptions.ServiceUnavailable: Failed to establish connection to ('127.0.0.1', MYPORTNO) (reason [Errno 111] Connection refused)

If I restart apache it works again, so currently as a bandaid, I'm doing that on a cron job every few hours, but that only seems to partially work.

I increased the page cache and heap calculating according to this article .

I increased the Open Files Limit from the default 1024 to 50000.

I doubled the server spec.

Structurally I create a database driver instance in a config file:

Config file:

db = None
def getDB():
    global db
    if not db:
        db = GraphDatabase.driver("bolt://localhost:XXXXXX", auth=basic_auth(DATABASE_USERNAME, DATABASE_PASSWORD), encrypted=False) 

    return db

I can then make queries in my module files as follows. I believe the following syntax properly manages connections being opened and closed due to the with statement, and so shouldn't leave open connections:

Module file:

db = config.getDB()

with db.session() as s:
        with s.begin_transaction() as tx:
            tx.run("the cypher", {the params})

I am also seeing the following error in the logs, although I'm not sure they are directly related.

ValueError: filedescriptor out of range in select()

Does anyone know what is causing this or how to fix it please? Thanks!

Answer 1

Turns out that I had a cron job that was running every 10 minutes and was leaving the driver instance open each time. Schoolboy error.