SQLAlchemy with asyncpg cannot perform operation: another operation is in progress

Question

我正在尝试以异步方式将 SQLAlchemy 与拖延任务执行器一起使用。

当我执行任务时：

@procrastinate.task(name='application__remove_components')
async def remove_applicatoin_components(application_id: int):
    """
    Uninstalls application components. If all application components uninstalled
    successfully runs post-terminate hooks.
    """
    async with session_maker() as session:
        application_manager = get_application_manager(session)
        application = await application_manager.get_application(application_id)
        manifest: TemplateSchema = load_template(application.manifest)
        try:
            await asyncio.gather(*[
                application_manager.uninstall_component(application, component)
                for component in manifest.components if component.enabled
            ])
        except ApplicationComponentUninstallException:
            await application_manager.set_state_status(application, ApplicationStatuses.error)
            raise

    await execute_post_terminate_hooks.defer_async(application_id=application_id)

我收到下一个错误：

task-executor_1  | /home/app/hub/crud/base.py:34: SAWarning: Usage of the 'Session.add()' operation is not currently supported within the execution stage of the flush process. Results may not be consistent.  Consider using alternative event listeners or connection-level operations instead.
task-executor_1  |   self.session.add(instance)
task-executor_1  | ERROR:procrastinate.worker.worker:Job application__remove_components[23373](application_id=78) ended with status: Error, lasted 2.818 s
task-executor_1  | Traceback (most recent call last):
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 739, in commit
task-executor_1  |     self.await_(self._transaction.commit())
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
task-executor_1  |     return current.driver.switch(awaitable)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
task-executor_1  |     value = await result
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/asyncpg/transaction.py", line 211, in commit
task-executor_1  |     await self.__commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/asyncpg/transaction.py", line 179, in __commit
task-executor_1  |     await self._connection.execute(query)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 317, in execute
task-executor_1  |     return await self._protocol.query(query, timeout)
task-executor_1  |   File "asyncpg/protocol/protocol.pyx", line 323, in query
task-executor_1  |   File "asyncpg/protocol/protocol.pyx", line 707, in asyncpg.protocol.protocol.BaseProtocol._check_state
task-executor_1  | asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress
task-executor_1  | 
task-executor_1  | The above exception was the direct cause of the following exception:
task-executor_1  | 
task-executor_1  | Traceback (most recent call last):
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1089, in _commit_impl
task-executor_1  |     self.engine.dialect.do_commit(self.connection)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 686, in do_commit
task-executor_1  |     dbapi_connection.commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 741, in commit
task-executor_1  |     self._handle_exception(error)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 682, in _handle_exception
task-executor_1  |     raise translated_error from error
task-executor_1  | sqlalchemy.dialects.postgresql.asyncpg.AsyncAdapt_asyncpg_dbapi.InterfaceError: <class 'asyncpg.exceptions._base.InterfaceError'>: cannot perform operation: another operation is in progress
task-executor_1  | 
task-executor_1  | The above exception was the direct cause of the following exception:
task-executor_1  | 
task-executor_1  | Traceback (most recent call last):
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/procrastinate/worker.py", line 231, in run_job
task-executor_1  |     task_result = await task_result
task-executor_1  |   File "/home/app/hub/services/procrastinate/tasks/application/terminate_flow.py", line 63, in remove_applicatoin_components
task-executor_1  |     await asyncio.gather(*[
task-executor_1  |   File "/home/app/hub/managers/applications.py", line 380, in uninstall_component
task-executor_1  |     await self.helm_manager.uninstall_release(
task-executor_1  |   File "/home/app/hub/managers/helm/manager.py", line 482, in uninstall_release
task-executor_1  |     await self.event_manager.create(EventSchema(
task-executor_1  |   File "/home/app/hub/managers/events.py", line 26, in create
task-executor_1  |     await self.db.create(event.dict())
task-executor_1  |   File "/home/app/hub/crud/base.py", line 35, in create
task-executor_1  |     await self.session.commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/session.py", line 582, in commit
task-executor_1  |     return await greenlet_spawn(self.sync_session.commit)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
task-executor_1  |     result = context.throw(*sys.exc_info())
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
task-executor_1  |     self._transaction.commit(_to_root=self.future)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 846, in commit
task-executor_1  |     return self._parent.commit(_to_root=True)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 836, in commit
task-executor_1  |     trans.commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2459, in commit
task-executor_1  |     self._do_commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2649, in _do_commit
task-executor_1  |     self._connection_commit_impl()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2620, in _connection_commit_impl
task-executor_1  |     self.connection._commit_impl()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1091, in _commit_impl
task-executor_1  |     self._handle_dbapi_exception(e, None, None, None, None)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
task-executor_1  |     util.raise_(
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
task-executor_1  |     raise exception
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1089, in _commit_impl
task-executor_1  |     self.engine.dialect.do_commit(self.connection)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 686, in do_commit
task-executor_1  |     dbapi_connection.commit()
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 741, in commit
task-executor_1  |     self._handle_exception(error)
task-executor_1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 682, in _handle_exception
task-executor_1  |     raise translated_error from error
task-executor_1  | sqlalchemy.exc.InterfaceError: (sqlalchemy.dialects.postgresql.asyncpg.InterfaceError) <class 'asyncpg.exceptions._base.InterfaceError'>: cannot perform operation: another operation is in progress
task-executor_1  | (Background on this error at: https://sqlalche.me/e/14/rvf5)

此代码是开源的，您可以根据需要进行调查。 我试过的：

1. 通过将NullPool传递给引擎来禁用连接池。
2. 由于实验试图将连接池大小扩展到 pool_size=50，max_overflow=100。
3. 试图创建引擎的新实例，session 制造商，session。

Answer 1

经过一些挖掘后，我发现 SQLAchemy session 在异步模式下运行不佳。 我曾经使用asyncio.gather来加快执行速度，但这导致同时创建 2 个或更多记录。

作为解决方法：

• 我已经删除了所有asyncio.gather并for s 的简单同步替换它们。
• 我用作用域异步 session 替换了简单的异步 session：

from asyncio import current_task
from typing import AsyncGenerator

from sqlalchemy.ext.asyncio import AsyncEngine
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.ext.asyncio import async_scoped_session
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.orm import sessionmaker

from core.configuration import settings


def get_engine(database_url: str = settings.DATABASE_URL) -> AsyncEngine:
    return create_async_engine(database_url, pool_pre_ping=True)


def get_session_maker(engine: str):
    return sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)


engine = get_engine()
session_maker = async_scoped_session(get_session_maker(engine), scopefunc=current_task)


async def get_session() -> AsyncGenerator[AsyncSession, None]:
    async with session_maker() as session:
        yield session

• 我已将每个经理的公共 session 替换为个人。

我不能将此答案标记为解决方案，因为我认为它存在重大缺陷。