[英]Multiple Python-Celery Scripts conflict and doesn't execute
I have two different python scripts in different directories that has celery schedulers. 我在具有celery调度程序的不同目录中有两个 不同的 python 脚本 。
Script 1: 脚本1:
import requests
from celery import Celery
from celery.schedules import crontab
import subprocess
celery = Celery()
celery.conf.enable_utc = False
@celery.task()
def proxy():
response = requests.get(url="XYZ")
proxies = response.text
paid_proxies = open("paid_proxies.txt", "w+")
paid_proxies.write(proxies.strip())
paid_proxies.close()
celery.conf.beat_schedule = {
"proxy-api": {
"task": "scheduler1.proxy",
"schedule": crontab(minute="*/5")
}
}
Commands that I use for executing it: 我用来执行的命令:
celery beat -A scheduler1.celery
celery worker -A scheduler1.celery
Script 2: 脚本2:
from celery import Celery
from celery.schedules import crontab
import subprocess
celery = Celery()
celery.conf.enable_utc = False
@celery.task()
def daily():
subprocess.run(["python3", "cross_validation.py"])
celery.conf.beat_schedule = {
"daily-scraper": {
"task": "scheduler2.daily",
"schedule": crontab(day_of_week="*", hour=15, minute=23)
}
}
Commands that I use for executing it: 我用来执行的命令:
celery beat -A scheduler2.celery
celery worker -A scheduler2.celery
The issues is when I execute Script 1, it works perfectly. 问题是当我执行脚本1时,它可以完美运行。 But when I try to execute Script 2, I get this error as Scheduler2 tries to execute tasks of scheduler1: 但是,当我尝试执行脚本2时,由于Scheduler2尝试执行scheduler1的任务而出现此错误:
[2019-09-14 15:10:00,127: ERROR/MainProcess] Received unregistered task of type 'scheduler1.proxy'. [2019-09-14 15:10:00,127:错误/ MainProcess]收到类型为“ scheduler1.proxy”的未注册任务。 The message has been ignored and discarded. 该消息已被忽略并丢弃。
Did you remember to import the module containing this task? 您还记得导入包含此任务的模块吗? Or maybe you're using relative imports? 也许您正在使用相对进口?
Please see http://docs.celeryq.org/en/latest/internals/protocol.html for more information. 有关更多信息,请参见http://docs.celeryq.org/en/latest/internals/protocol.html 。
The full contents of the message body was: '[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b) Traceback (most recent call last): File "/home/PycharmProjects/data_scraping/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 559, in on_task_received strategy = strategies[type_] KeyError: 'scheduler1.proxy' 邮件正文的完整内容为:'[[],{},{“ callbacks”:null,“ errbacks”:null,“ chain”:null,“ chord”:null}]'(77b)追溯(大部分最近一次通话结束):文件“ /home/PycharmProjects/data_scraping/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py”,行559,在on_task_received strategy = strategys [type_] KeyError中: 'scheduler1.proxy'
I tried referring multiple answers but nothing worked. 我尝试引用多个答案,但没有任何效果。
The problem that you are seeing is that celery is using the same "broker" in both project 1 and project 2. In order to use two different celery projects simultaneously, all you have to do is give them different brokers. 您看到的问题是,芹菜在项目1和项目2中都使用相同的“经纪人”。为了同时使用两个不同的芹菜项目,您要做的就是给他们提供不同的经纪人。 You can specify a broker using the broker_url
setting . 您可以使用broker_url
设置指定代理。
We typically use redis as a broker, so it is very simple to put one project on redis db 0 and the other project on redis db 1. That said, there is a lot of thinking that normally goes into which broker to use , and deciding on a broker is outside the scope of this particular question. 我们通常使用redis作为代理,因此将一个项目放在redis db 0上,而将另一个项目放在redis db 1上非常简单。也就是说,有很多想法通常会涉及使用哪个代理 ,并决定关于经纪人的问题不在此特定问题的范围之内。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.