使用 Celery 和 Gino 保存 object
[英]Save an object using Celery and Gino
问题描述
I have the following pipeline
我有以下管道
Event model (based on Gino 'db' object):事件 model（基于 Gino 'db' 对象）：
class Event(db.Model):
__tablename__ = "events"
id = db.Column(db.BigInteger(), primary_key=True)
request_timestamp = db.Column(db.DateTime(), nullable=False)
service = db.Column(db.String(), nullable=False)
url = db.Column(db.String(), nullable=False)
status_code = db.Column(db.Integer(), nullable=False)
response_time = db.Column(db.DateTime(), nullable=False)
FastApi app has a POST view with call to celery task to pass the data of the Event object: FastApi 应用程序有一个 POST 视图，调用 celery 任务来传递事件 object 的数据：
@router.post("/events/add")
async def add_event(event: EventModel):
event_data = {'request_timestamp': event.request_timestamp.replace(tzinfo=None),
'service': event.service,
'url': event.url,
'status_code': event.status_code,
'response_time': event.response_time.replace(tzinfo=None)
}
task = celery_app.send_task("monitoring_service.src.monitoring_service.worker.celery_worker.add_to_db",
kwargs=event_data)
return JSONResponse(content="Event recorded successfully", status_code=200)
And celery worker which must save an Event object to database:并且 celery 工作人员必须将事件 object 保存到数据库：
@celery_app.task(acks_late=True)
async def add_to_db(request_timestamp, service, url, status_code, response_time):
event = await Event.create(
request_timestamp=datetime.strptime(request_timestamp, '%Y-%m-%dT%H:%M:%S.%f'),
service=service,
url=url,
status_code=status_code,
response_time=datetime.strptime(response_time, '%Y-%m-%dT%H:%M:%S.%f'),
)
return {"status": True}
Now I'm receiving an error during object saving process:现在我在 object 保存过程中收到错误消息：
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 479, in trace_task
mark_as_done(
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 158, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 442, in store_result
self._store_result(task_id, result, state, traceback,
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/__init__.py", line 51, in _inner
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/__init__.py", line 130, in _store_result
session.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
_emit_update_statements(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 994, in _emit_update_statements
c = cached_connections[connection].execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1207, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1205, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 858, in _init_compiled
param = dict(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 861, in <genexpr>
processors[key](compiled_params[key])
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 1689, in process
value = dumps(value, protocol)
sqlalchemy.exc.SQLAlchemyError: (builtins.TypeError) cannot pickle 'coroutine' object
warn(RuntimeWarning(
[2021-02-19 10:19:39,010: ERROR/ForkPoolWorker-1] Task monitoring_service.src.monitoring_service.worker.celery_worker.add_to_db[994ae754-f216-4fa9-b502-5fbb0221011c] raised unexpected: SQLAlchemyError("(builtins.TypeError) cannot pickle 'coroutine' object")
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 479, in trace_task
mark_as_done(
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 158, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 442, in store_result
self._store_result(task_id, result, state, traceback,
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/__init__.py", line 51, in _inner
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/__init__.py", line 130, in _store_result
session.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
_emit_update_statements(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 994, in _emit_update_statements
c = cached_connections[connection].execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1207, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1205, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 858, in _init_compiled
param = dict(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 861, in <genexpr>
processors[key](compiled_params[key])
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 1689, in process
value = dumps(value, protocol)
sqlalchemy.exc.SQLAlchemyError: (builtins.TypeError) cannot pickle 'coroutine' object
If I call event = await Event.create(...) inside a view it is successfull and saves data to db.如果我在视图中调用event = await Event.create(...)它是成功的并将数据保存到数据库。 If I delete 'async/await' from Celery worker, this error doesn't show up, Celery task has 'SUCCESS' state but database is empty.如果我从 Celery 工作人员中删除 'async/await'，则不会出现此错误，Celery 任务有 'SUCCESS' state 但数据库为空。 Can't really understand what is not right.无法真正理解什么是不正确的。
1 个解决方案
解决方案1
0 已采纳 2021-02-19 16:16:41
Simple: currently, Celery is not able to handle asynchronous functions as tasks.简单：目前，Celery 无法将异步函数作为任务处理。 You need to wrap it with asyncio.run (https://docs.python.org/3/library/asyncio-task.html#asyncio.run ):您需要用asyncio.run （https://docs.python.org/3/library/asyncio-task.html#asyncio.run ）包装它：
async def add_to_db(request_timestamp, service, url, status_code, response_time):
event = await Event.create(
request_timestamp=datetime.strptime(request_timestamp, '%Y-%m-%dT%H:%M:%S.%f'),
service=service,
url=url,
status_code=status_code,
response_time=datetime.strptime(response_time, '%Y-%m-%dT%H:%M:%S.%f'),
)
@celery_app.task(acks_late=True)
def add_to_db_task(request_timestamp, service, url, status_code, response_time):
asyncio.run(add_to_db(request_timestamp, service, url, status_code, response_time))
return {"status": True}
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
在 Django 中使用 Celery：如何在任务中保存对象？ - Using Celery with Django: How do you save an object inside of a task?
在Celery任务期间保存对象 - Save an object during a Celery task
多个Django Celery Tasks尝试保存到同一对象并失败 - Multiple Django Celery Tasks are trying to save to the same object and failing
在Celery任务中保存对象后，Django匹配查询不存在 - Django matching query does not exist after object save in Celery task
使用django在芹菜中传递User对象 - Passing User object in celery using django
使用Celery（python）和RabbitMQ将任务结果保存到特定文件位置？ - Using Celery (python) & RabbitMQ to save results of tasks to a specific file location?
我想使用 gino 和 asyncpg 将我的应用程序连接到 heroku Postgress 数据库 - I want to connect my app to a heroku Postgress database using gino and asyncpg
我在尝试使用 pytest for FastAPI + Gino 应用程序运行测试时遇到一个奇怪的错误 - I'm getting a weird error trying to run tests using pytest for FastAPI + Gino app
Django Celery-使用RabbitMQ在视图之间以及任务之间传递对象 - Django Celery - Passing an object to the views and between tasks using RabbitMQ
使用 Python 3 时，Celery 'module' 对象没有属性 'app' - Celery 'module' object has no attribute 'app' when using Python 3
相关标签