Laravel horizon queued jobs fails with MaxAttemptsExceededException

Question

I dispatch jobs to do some work on my horizon queue. It appears the same jobs are always failing with 60s runtime, which to me looks like a timeout issue:

I dispatch around 10 jobs via a cron schedule, and these 4 are always failing.

This is my config/horizon.php (just the interesting parts):

<?php

use Illuminate\Support\Str;

return [

    'waits' => [
        'redis:default' => 60,
    ],

    'memory_limit' => 512,

    'defaults' => [
        'my-app-queue' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'auto',
            'minProcesses' => 1,
            'maxProcesses' => 1,
            'balanceMaxShift' => 1,
            'balanceCooldown' => 3,
            'memory' => 512,
            'tries' => 3,
            'nice' => 0,
            'timeout' => 300,
        ],
    ],

    'environments' => [
        'production' => [
            'my-app-queue' => [
                'maxProcesses' => 10,
            ],
        ],

        'local' => [
            'my-app-queue' => [
                'maxProcesses' => 10,
            ],
        ],
    ],
];

As you can see, I have set a timeout of 300 seconds, but the job always failing at 60 second mark.

I start the horizon queue processor in my docker container with this entrypoint command: php /path/to/artisan horizon

To debug the issue, I instantiated the job class manually to see where it is failing like this:

    $test = new DownloadBlockedIPFeed(8);
    $test->handle();
    exit('Done');

This does seem to be taking longer than 60 seconds, but it does complete.

So my question is - how do you properly set the timeout for laravel horizon?

I am using laravel 8.x and latest version of the horizon package.

Answer 1

It may also be the balance strategy. I've found that If I set the 'balance' option to 'auto', it gives the MaxAttemptsExceededException errors. changing it to 'simple' or 'false' seems to be the solution.

Not sure if this is a bug, but I don't think this is expected behavior either. I've seen jobs that where executed (logging in the handle method of the job) but killed off in the middle for retry.

Answer 2

Check the value for --timeout= in the spawned horizon:supervisor process. If this is 60 seconds, adjust the timeout value in config/horizon.php .

Also check retry_after in config/queue.php .