Laravel 6 Horizon Slow - long wait time

Question

I am using https://laravel.com/docs/6.x/horizon to manage the queue on my application (using redis backend).

The jobs on the queue are ephemeral - each queued job is supposed to make an api call (to a third party) and write the response to db.

Today I noticed lots of jobs are stuck / processing very slowly:

I am running 10 queue workers with sufficient memory on the host node. This is my config/horizon.php

return [
    'domain' => null,
    'path' => 'horizon',
    'use' => 'default',
    'prefix' => env('HORIZON_PREFIX', 'horizon:'),
    'middleware' => ['web', 'basic.api.credential.auth'],
    'waits' => [
        'redis:default' => 60,
    ],
    'trim' => [
        'recent' => 60,
        'completed' => 60,
        'recent_failed' => 40320,
        'failed' => 40320,
        'monitored' => 40320,
    ],
    'fast_termination' => false,
    'memory_limit' => 512,
    'environments' => [
        'production' => [
            'queue-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'simple',
                'processes' => 10,
                'tries' => 2,
                'timeout' => 1800, // 30 Minutes
                'memory' => 512,
            ],
        ],
        // ...
    ],
];

I checked the remote api server to see if the bottleneck is there, but it's responding quite fast (less than 1 sec per api call). Checking the server, I can't see any load on it and the overall memory/cpu utilisation is quite low.

Laravel v6.18.37
PHP v7.3.21
Horizon v3.7.2

Any ideas what's causing such a huge slowdown? How do I debug this to find out what's going on?

I even tried restarting the server and it did not help. I am not seeing any timeouts / errors in the logs either.

Answer 1

The problem turned out to be on the database, a few table that we use for lookup was not properly optimised - after adding indexes, the speed improved tremendously.

Sorry false alarm.