compose 服务中的 psycopg2 有时 (<1%) 无法将主机名“db”转换为地址：名称解析中的临时故障

Question

我目前正在运行名为app的 Compose 服务的多个实例（ --scale app=40 ：计算机有 56 个 CPU 内核和 256 GB RAM），它嵌入了一个 Python 脚本，该脚本连接读/写数据到一个唯一的 PostgreSQL 数据库，已知作为db服务（此服务仅运行一次，并且在app服务之前启动，直到 postgres 告诉我数据库已准备好接受连接）来自相同的组合。

有时（很少，但这对我来说仍然太多），出现以下 psycopg2 错误：

psycopg2.OperationalError:
  could not translate host name "db" to address:
    Temporary failure in name resolution

这是完整的堆栈跟踪：

 Traceback (most recent call last):
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
     return fn()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
     return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
     fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
     rec = pool._do_get()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
     self._dec_overflow()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
     raise exception
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
     return self._create_connection()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
     return _ConnectionRecord(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
     self.__connect()
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
     pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
     raise exception
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
     self.dbapi_connection = connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 241, in <lambda>
     return lambda crec: creator()
   File "/app/scripts/database_connectors.py", line 112, in connector
     res = psycopg2.connect(**DB_PARAMETERS)
   File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
 psycopg2.OperationalError: could not translate host name "db" to address: Temporary failure in name resolution

我想知道为什么以及如何正确调试此错误或弄清楚到底发生了什么，因为我的db服务运行良好，并且多个app实例可以在 99% 的时间正确地到达它。

这不会同时弹出，因此可能不相关，但以防万一，此错误也会不时出现：

psycopg2.errors.DiskFull:
  could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
    No space left on device

或者：

sqlalchemy.exc.OperationalError:
  (psycopg2.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.3516559362" to 146703328 bytes:
    No space left on device

版本控制：

$ docker --version
Docker version 20.10.18, build b40c2f6

$ docker-compose --version
Docker Compose version v2.5.0

$ dc exec app bash

root@38dffec72240:/app# python3

Python 3.9.7 (default, Sep  3 2021, 02:02:37) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'

更多的：

$ df -h
Filesystem                              Size  Used Avail Use% Mounted on
(...)
/dev/mapper/vps-vg-root                 665G  355G  277G  57% /
(...)