laravel - run a task after a queue finish all its jobs

Question

I need to run a task for a list that can reach thousands of items.

To avoid a single job running for a long time, I've created a job to enqueue all items.

The problem is that I need to run another task before and after that queue gets done.

The solution I see is to use delays:

$schedule->job(new \App\Jobs\PauseSystem())     ->hourly('00:01');
$schedule->job(new \App\Jobs\EnqueueAllItems()) ->hourly('00:02'); // adds all items as separated job in a queue
$schedule->job(new \App\Jobs\ReopenSystem())    ->hourly('55:00');

This way I have the time between 00:02 and 55:00 to ensure all items have been completed.

It looks not safe and maybe causes job overlapping.

Is there any safer way to run a task after a queue finish all its jobs?

Answer 1

Since you mentioned there are parallel and multiple jobs, here is how we solved a similar issue. We have a reporting system to prepare a report from multiple customers. There were more than 1000 customers.

• Each ReportPusher job was responsible for a single customer
• In this job we were getting report from different databases and pushing all the data to a bucket(redis list).
• If all the jobs(%99 of the jobs sometimes) are completed to work, another ReportCollector job should do its work.
• Gets all the data from the single bucket, formats, creates an excel and sends an email. - This collector job has to run after all the ReportPusher jobs completed.

Here how we did;

• Trigger all ReportPusher jobs at the same time
• Set total number of triggered jobs in some place (redis key for example) $total
• Trigger ReportCollector n minutes later (it could be 15 minutes)
• Each ReportPusher jobs increments another key when they complete their process $incremented

• 15 Minutes later when ReportCollector works, do these;

  • If total equals to incremented count let ReportCollector does it work
  • If not, trigger the same job with t delay(you decide) + incremented attempt
  • After n attempts(you decide) if the counts still don't match(the %99 i mentioned above) calculate the report.

This fallback policy is to prevent total failure if there is a mistake in one/two of the customers. We don't wont to throw all the calculated data(%99 of the customers) just because of one/two customers. You may put some alert/error tracking system to fix corrupted data later.