python中celery组任务全部完成后如何调用方法

Question

Currently I am doing the celery group task and I want to call upload_local_directory(output_file)method after all the tasks completed. I tried below approach but its not waiting the job to be completed.

tasks = [make_tmp_files.s(page.object_list, path + str(uuid.uuid4() + '.csv')) for page in paginator]
job = group(tasks)
job.apply_async()
job.get()

output_file = 'final.zip'
upload_local_directory_into_S3(output_file)

make_tmp_files method is the celery job method.

“backend” also defined in the celery object.

please comment If more information needed.

Answer 1

You either chain your group and the final, make_tmp_files task, or you use Chord to accomplish the same. Do not be alarmed if you see that Celery automatically converts group+task chain into a Chord.

Answer 2

Within celery there is a difference between a group of tasks and a group of task results. If you were to change your code to the following it should work:

tasks = [make_tmp_files.s(page.object_list, path + str(uuid.uuid4() + '.csv')) for page in paginator]
job = group(tasks)
job_results = job.apply_async()
job_results.get()

output_file = 'final.zip'
upload_local_directory_into_S3(output_file)

job.apply_async() returns a group of AsyncResults. As a user, you need to check the result of AsyncResult, not the tasks themselves.

Reference: https://docs.celeryproject.org/en/stable/userguide/canvas.html#groups

Hopefully that helps!