在使用芹菜的简单任务中表现不佳

Question

I am currently facing subpar performance when executing the following usecase:

I have two files - tasks.py

# tasks.py
from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//', backend='rpc://',worker_prefetch_multiplier=1)

@app.task
def task(array_of_elements):
    return [x ** 2 for x in array_of_elements]

and run.py

# run.py
from celery import group
from itertools import chain, repeat
from tasks import task
import time

def grouper(n, iterable, padvalue=None):
    return zip(*[chain(iterable, repeat(padvalue, n-1))]*n)

def fun1(x):
    return x ** 2

if __name__ == '__main__':
    start = time.time()
    items = [list(x) for x in grouper(10000, range(10000))]
    x = group([task.s(item) for item in items])
    r = x.apply_async()
    d = r.get()
    end = time.time()
    print(f'>celery: {end-start} seconds')

    start = time.time()
    res = [fun1(x) for x in range(10000)]
    end = time.time()
    print(f'>normal: {end-start} seconds')

When I am trying running celery: celery -A tasks worker --loglevel=info

and trying to run:

python run.py

This is the output I get:

>celery: 0.19174742698669434 seconds
>normal: 0.004475116729736328 seconds

I have no idea why the performance is worse in celery?

I am trying to understand how can I achieve map-reduce paradigm using celery like split a huge array into smaller chunks, do some processing and bring results back

Am I missing some critical configuration?

Answer 1

Map-reduce paradigm is not supposed to be faster but to be better at scaling.

There is always an overhead for a MR job compare to a local running job implementing the same computation : process scheduling, communication, shuffling, etc.

Your benchmark is not relevant because MR and local run are either approaches, depending on the data set size. At some point you swap from a local running approach to a MR approach because your dataset become too large for one node.