使用 Celery，在单个 celery 任务中执行 URL 列表。 可能吗？

Question

views.py

urls=["https//:.....com,https//:.....com,etc.."]
for i in urls:
   r=process.delay(i)

When I'm calling the celery task it execute separate task. How to execute the set of lists in a single celery task?

tasks.py

@app.task
def process(url):
    r = requests.get(url, allow_redirects=True)
    return r

Answer 1

You can pass a group of URLs or even the whole list to the celery task. But this doesn't make much sense as we haven't actually taken advantage of the parallelized outgoing requests since each URL would be accessed serially, one after the other.

views.py

urls=["https//:.....com", "https//:.....com", ...]
r = process.delay(urls)

tasks.py

@app.task
def process(urls):
    results = []
    for url in urls:
        r = requests.get(url, allow_redirects=True)
        results.append(r)
    return results