Laravel Queue with Supervisor not working with Multiple Websites

Question

When adding a single website running a Laravel Queue to Supervisor, delayed jobs are processed as they should.

When I add another website to Supervisor, delayed jobs are not processed for any of the websites/queues.

I have tried rereading, reloading, and restarting the service. The supervisord.log shows all the processes running, but jobs are not being triggered.

If I stop all but 1 worker, the queue works.

Laravel 5.7

Supervisord 3.3.1

redis-cli 3.2.6

Debian/Nginx

sudo nano /etc/supervisor/conf.d/website-a-worker.conf

[program:website-a-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/website.a/artisan queue:work redis --sleep=3 --tries=3
autostart=true
autorestart=true
#user=laravel
numprocs=8
autostart=true
autorestart=true
stdout_logfile=/var/www/website.a/storage/logs/worker.log
stderr_logfile=/var/www/website.a/storage/logs/worker.err.log

sudo nano /etc/supervisor/conf.d/website-b-worker.conf

[program:website-b-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/website.b/artisan queue:work redis --sleep=3 --tries=3
autostart=true
autorestart=true
#user=laravel
numprocs=8
autostart=true
autorestart=true
stdout_logfile=/var/www/website.b/storage/logs/worker.log
stderr_logfile=/var/www/website.b/storage/logs/worker.err.log

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start website-a-worker:*
sudo supervisorctl start website-b-worker:*

// config/queue.php
...
'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 600,
            'block_for' => null,
        ],
...

<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;

class SomeJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function handle()
    {
       \Log::info('This only gets triggered if 1 supervisor worker is running.');
    }
}

> sudo service supervisor restart
> cd /var/www/website.a
> php artisan tinker

>>> App\Jobs\SomeJob::dispatch()->delay(1);
# Job does not get triggered
>>> exit

The job never gets triggered. However, if I stop supervisor for website.b, website.a will process jobs.

> sudo supervisorctl stop website-b-worker:*
website-b-worker:website-b-worker_02: stopped
website-b-worker:website-b-worker_03: stopped
website-b-worker:website-b-worker_00: stopped
website-b-worker:website-b-worker_01: stopped
website-b-worker:website-b-worker_06: stopped
website-b-worker:website-b-worker_07: stopped
website-b-worker:website-b-worker_04: stopped
website-b-worker:website-b-worker_05: stopped

> php artisan tinker

>>> App\Jobs\SomeJob::dispatch()->delay(1);
# Job gets triggered!

How can I run multiple workers (at least 1 for each website)?

Answer 1

I've used Supervisor with Laravel and Redis but only for 1 website at a time. However I did some research and I found that you can specify the name of the Queue process in Laravel https://laravel.com/docs/5.8/queues#connections-vs-queues .

A Laravel Job has a setter method onQueue() to specificy name to a queue. laravel docs example :

Job::dispatch()->onQueue('emails');

But in your code SomeJob you can call it directly with $this->onQueue('queue_name') ; your code can be like :

<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;

class SomeJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function handle()
    {
        // on website a
        $this->onQueue('websiteAQueue');
        \Log::info('This only gets triggered if 1 supervisor worker is running.');
    }
}

And in Supervisor process for Website A, specify the queue parameter to be websiteAQueue :

[program:website-a-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/website.a/artisan queue:work redis --queue=WebsiteAQueue --sleep=3 --tries=3
autostart=true
autorestart=true
#user=laravel
numprocs=8
autostart=true
autorestart=true
stdout_logfile=/var/www/website.a/storage/logs/worker.log
stderr_logfile=/var/www/website.a/storage/logs/worker.err.log

This way when Laravel is sending the Job, it will send it to that specified Queue and not the default one it found.

EDIT------------------------

Or better, you can use a different queue name for each Website config :

// Website A : config/queue.php
...
'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'websiteAQueue',
            'retry_after' => 600,
            'block_for' => null,
        ],
...

This way all the queues are sent to websiteAQueue automatically, so you don't have to use $this->onQueue() on each Job.